Image decoding apparatus, image encoding apparatus, and method and program for image decoding and encoding

ABSTRACT

An encoded bit stream is processed by a lossless decoding unit ( 52 ), an inverse quantization unit ( 53 ), and an inverse orthogonal transform unit ( 54 ) in this order, to obtain orthogonally transformed coefficient data and encoding parameter information. The inverse orthogonal transform unit ( 54 ) performs an inverse orthogonal transform on the coefficient data by using bases that are set beforehand in accordance with the locations of transform blocks in a macroblock indicated by the encoding parameter information. In this manner, prediction error data is obtained. An intra prediction unit ( 62 ) generates predicted image data. An addition unit ( 55 ) adds the predicted image data to the prediction error data, to decode image data. By using bases that are set in accordance with the locations of transform blocks, an optimum inverse orthogonal transform can be performed, and encoding efficiency can be increased.

TECHNICAL FIELD

The present invention relates to an image decoding apparatus, an imageencoding apparatus, and a method and a program for image decoding andencoding. More particularly, the present invention provides an imagedecoding apparatus and an image encoding apparatus that are capable ofperforming efficient decoding and encoding, and a method and a programfor such decoding and encoding.

BACKGROUND ART

In recent years, to handle image information as digital information andachieve high-efficiency information transmission and accumulation,apparatuses compliant with a standard, such as MPEG for compressingimage information through orthogonal transforms and motion compensationsby using redundancy inherent to image information, have been spreadingboth among broadcast stations to distribute information and amonggeneral households to receive information.

Particularly, MPEG2 (ISO/IEC 13818-2) is defined as a general-purposeimage encoding technique. The MPEG2 compression technique is applicableto interlaced images and non-interlaced images, and tostandard-resolution images and high-definition images. Currently, MPEG2is used for a wide range of applications for professionals and generalconsumers. By using the MPEG2 compression technique, a bit rate of 18 to22 Mbps is assigned to interlaced images with a high resolution of1920×1088 pixels, for example, to achieve high compression rates andexcellent image quality.

MPEG2 is designed mainly for high-quality image encoding forbroadcasting, but is not compatible with lower bit rates than MPEG1 orencoding techniques with higher compression rates. As mobile terminalsare becoming popular, the demand for such encoding techniques isexpected to increase in the future, and to meet the demand, the MPEG4encoding technique has been set as a standard. As for image encodingtechniques, the ISO/IEC 14496-2 standard was approved as aninternational standard in December 1998.

Further, in recent years, H.264 and MPEG-4 Part 10 (Advanced VideoCoding, hereinafter referred to as H.264/AVC), which can achieve ahigher encoding efficiency than encoding techniques such as MPEG2 andMPEG4, have become an international standard, though requiring a largeramount of calculation for decoding. H.264/AVC is based on H.26L, butalso has functions unsupported by H.26L.

Patent Document 1 and the like disclose more efficient image dataencoding using H.264/AVC.

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No.    2008-4984

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

In intra predictions, there has been a suggested technique called MDDT(Mode Dependent Directional Transform) that switches transform methodsin accordance with the directions of the intra predictions. Where theMDDT technique is used, it is difficult to increase encoding efficiency,unless transforms to be performed in accordance with the directions ofintra predictions are optimized.

Therefore, the present invention aims to provide an image decodingapparatus and an image encoding apparatus that can increase the encodingefficiency, and methods and programs for image decoding and encoding.

Solution to Problems

A first aspect of the invention is an image decoding apparatus thatperforms an orthogonal transform on prediction error data of eachtransform block, and decodes image data from an encoded bit streamgenerated by processing coefficient data subjected to the orthogonaltransform, the prediction error data being the difference between theimage data and predicted image data. The image decoding apparatusincludes: a data processing unit that processes the encoded bit streamto obtain the coefficient data subjected to the orthogonal transform andencoding parameter information; an inverse orthogonal transform unitthat performs an inverse orthogonal transform on the coefficient data byusing a base to obtain the prediction error, the base being setbeforehand in accordance with the location of the transform block in amacroblock indicated by the encoding parameter information; a predictedimage data generation unit that generates the predicted image data; andan addition unit that adds the predicted image data generated by thepredicted image data generation unit to the prediction error obtained bythe inverse orthogonal transform unit, to decode the image data.

In the image decoding apparatus of the invention, when an inverseorthogonal transform is performed on orthogonally transformedcoefficient data obtained by processing an encoded bit stream, aninverse orthogonal transform such as a Karhunen-Loeve inverse transformis performed by using a base that is set beforehand in accordance withthe block locations of transform blocks in a macroblock indicated byencoding parameter information that is contained in the encoded bitstream and is designed for decoding image data, or with the blocklocations and the prediction mode indicated by the encoding parameterinformation. When the macroblock contains more than one transform block,a Karhunen-Loeve inverse transform is performed by using a base onorthogonally transformed coefficient data of the block using thelowest-frequency component coefficient after the orthogonal transform ofthe respective transform blocks, the base being set beforehand inaccordance with the prediction mode. The base used by the inverseorthogonal transform unit is an inverse matrix of the base used in theorthogonal transform of the prediction error data of each transformblock. With such bases being provided in advance, an inverse orthogonaltransform is performed by using a base selected in accordance with ablock location or the like, to generate the prediction error data notyet subjected to the orthogonal transform.

A second aspect of the invention is an image decoding method forperforming an orthogonal transform on prediction error data of eachtransform block, and decoding image data from an encoded bit streamgenerated by processing coefficient data subjected to the orthogonaltransform, the prediction error data being the difference between theimage data and predicted image data. The image decoding method includesthe steps of: processing the encoded bit stream to obtain thecoefficient data subjected to the orthogonal transform and encodingparameter information; performing an inverse orthogonal transform on thecoefficient data by using a base to obtain the prediction error, thebase being set beforehand in accordance with the location of thetransform block in a macroblock indicated by the encoding parameterinformation; generating the predicted image data; and adding thegenerated predicted image data to the prediction error obtained by theinverse orthogonal transform unit, to decode the image data.

A third aspect of the invention is a program for causing a computer toperform an orthogonal transform on prediction error data of eachtransform block, and decode image data from an encoded bit streamgenerated by processing coefficient data subjected to the orthogonaltransform, the prediction error data being the difference between theimage data and predicted image data. The program causes the computer to:process the encoded bit stream to obtain the coefficient data subjectedto the orthogonal transform and encoding parameter information; performan inverse orthogonal transform on the coefficient data by using a baseto obtain the prediction error, the base being set beforehand inaccordance with the location of the transform block in a macroblockindicated by the encoding parameter information; generate the predictedimage data; and add the generated predicted image data to the predictionerror obtained by the inverse orthogonal transform unit, to decode theimage data.

A fourth aspect of the invention is an image encoding apparatus thatencodes image data. The image encoding apparatus includes: a predictionunit that generates predicted image data of the image data; asubtraction unit that generates prediction error data, the predictionerror data being the difference between the image data and the predictedimage data; an orthogonal transform unit that performs an orthogonaltransform on the prediction error of each transform block by using abase set beforehand in accordance with a location of the transform blockin a macroblock; and a data processing unit that processes data outputfrom the orthogonal transform unit, to generate an encoded bit stream.

In the image encoding apparatus of the invention, when prediction errordata indicating the difference between image data and predicted imagedata is orthogonally transformed for each transform block, an orthogonaltransform such as a Karhunen-Loeve transform is performed by using abase set beforehand in accordance with the block locations of thetransform blocks in a macroblock or with the block locations and theprediction mode used when the predicted image data is generated. Whenthe macroblock contains more than one transform block, a Karhunen-Loevetransform is performed on the block formed with the lowest-frequencycomponent coefficient after the orthogonal transform of the respectivetransform blocks. In this Karhunen-Loeve transform, a base that is setbeforehand in accordance with the prediction mode is used. This base iseigenvectors corresponding to the eigenvalues of a matrix calculatedfrom each macroblock size, each transform block size, each location ofthe transform blocks in the macroblock, and prediction error data ineach transform block in each prediction mode. The base is grouped inaccordance is the distance between bases or the distance from areference pixel. With such bases being provided in advance, anorthogonal transform is performed by using a base selected in accordancewith a block location or the like. Further, processing such asquantization and lossless coding is performed on the orthogonallytransformed coefficient data, to generate an encoded bit stream.

A fifth aspect of the invention is an image encoding method for encodingimage data. The image encoding method includes the steps of: generatingpredicted image data of the image data; generating prediction errordata, the prediction error data being the difference between the imagedata and the predicted image data; and performing an orthogonaltransform on the prediction error of each transform block by using abase set beforehand in accordance with the location of the transformblock in a macroblock.

A sixth aspect of the invention is a program for causing a computer toencode image data. The program causes the computer to: generatepredicted image data of the image data; generate prediction error data,the prediction error data being the difference between the image dataand the predicted image data; and perform an orthogonal transform on theprediction error of each transform block by using a base set beforehandin accordance with the location of the transform block in a macroblock.

The program of the present invention is a program that can be providedto a general-purpose computer system capable of executing variousprogram codes via a storage medium or a communication medium provided ina computer-readable form, such as an optical disk, a magnetic disk, asemiconductor memory, or a network. By providing such a program in acomputer-readable form, processing according to the program is realizedin a computer system.

Effects of the Invention

According to the invention, at the time of image data encoding, anorthogonal transform is performed by using a base that is set beforehandin accordance with the block locations of transform blocks in amacroblock. Also, in decoding an encoded bit stream generated byprocessing coefficient data obtained through an orthogonal transformusing a base set beforehand in accordance with the block locations, aninverse orthogonal transform is performed by using the base that is setbeforehand in accordance with the block locations in the macroblockindicated by the encoding parameter information contained in the encodedbit stream. Accordingly, the orthogonally transformed coefficient datacan be returned to the prediction error data not yet subjected to theorthogonal transform. As an orthogonal transform and an inverseorthogonal transform are performed by using the bases compatible withthe block locations in a macroblock as described above, transforms thatare optimized in accordance with the block locations can be performed,and encoding efficiency can be increased.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating the structure of an image encodingapparatus.

FIG. 2 are diagrams showing intra prediction modes for 4×4 pixel blocks.

FIG. 3 are diagrams showing the relationships between prediction modesand prediction errors.

FIG. 4 is a diagram illustrating KL transforms at an orthogonaltransform unit.

FIG. 5 is a diagram illustrating the structure of the orthogonaltransform unit.

FIG. 6 is a flowchart showing an image encoding operation.

FIG. 7 is a flowchart showing prediction operations.

FIG. 8 is a flowchart showing intra prediction operations.

FIG. 9 is a flowchart showing inter prediction operations.

FIG. 10 is a flowchart showing an encoding parameter generatingoperation.

FIG. 11 is a flowchart showing an orthogonal transform operation.

FIG. 12 are diagrams for explaining orthogonal transform operations.

FIG. 13 is a diagram illustrating the structure of an image decodingapparatus.

FIG. 14 is a diagram illustrating the structure of an inverse orthogonaltransform unit.

FIG. 15 is a flowchart showing an image decoding operation.

FIG. 16 is a flowchart showing an inverse orthogonal transformoperation.

FIG. 17 are diagrams for explaining inverse orthogonal transformoperations.

FIG. 18 is a flowchart showing prediction operations.

FIG. 19 is a flowchart showing a base learning operation.

FIG. 20 are diagrams for explaining base grouping.

FIG. 21 is a diagram schematically showing an example structure of atelevision apparatus.

FIG. 22 is a diagram schematically showing an example structure of aportable telephone device.

FIG. 23 is a diagram schematically showing an example structure of arecording/reproducing apparatus.

FIG. 24 is a diagram schematically showing an example structure of animaging apparatus.

MODES FOR CARRYING OUT THE INVENTION

The following is a description of embodiments for carrying out theinvention. The description will be made in the following order.

1. Structure of an Image Encoding Apparatus

2. Structure of the Orthogonal Transform Unit

3. Operations of the Image Encoding Apparatus

4. Structure of an Image Decoding Apparatus

5. Structure of the Inverse Orthogonal Transform Unit

6. Operations of the Image Decoding Apparatus

7. Base Learning Operation

8. Case of Software Processing

9. Case of Application to an Electronic Device

<1. Structure of an Image Encoding Apparatus>

FIG. 1 illustrates the structure of an image encoding apparatus. Theimage encoding apparatus 10 includes an analog/digital conversion unit(A/D conversion unit) 11, a picture rearrangement buffer 12, asubtraction unit 13, an orthogonal transform unit 14, a quantizationunit 15, a lossless coding unit 16, an accumulation buffer 17, and arate control unit 18. The image encoding apparatus 10 further includesan inverse quantization unit 21, an inverse orthogonal transform unit22, an addition unit 23, a deblocking filter 24, a frame memory 27, anintra prediction unit 31, a motion prediction/compensation unit 32, anda predicted image/optimum mode select unit 33.

The A/D conversion unit 11 converts analog image signals into digitalimage data, and outputs the image data to the picture rearrangementbuffer 12.

The picture rearrangement buffer 12 rearranges the frames of the imagedata output from the A/D conversion unit 11. The picture rearrangementbuffer 12 rearranges the frames in accordance with the GOP (Group ofPictures) structure related to encoding operations, and outputs therearranged image data to the subtraction unit 13, the intra predictionunit 31, and the motion prediction/compensation unit 32.

The subtraction unit 13 receives the image data output from the picturerearrangement buffer 12 and predicted image data selected by the laterdescribed predicted image/optimum mode select unit 33. The subtractionunit 13 calculates prediction error data that is the difference betweenthe image data output from the picture rearrangement buffer 12 and thepredicted image data supplied from the predicted image/optimum modeselect unit 33, and outputs the prediction error data to the orthogonaltransform unit 14.

The orthogonal transform unit 14 performs an orthogonal transformoperation on the prediction error data output from the subtraction unit13. When making an intra prediction, the orthogonal transform unit 14performs an orthogonal transform operation in accordance with theprediction mode. The orthogonal transform unit 14 outputs coefficientdata obtained by performing the orthogonal transform operation to thequantization unit 15.

The quantization unit 15 receives the coefficient data output from theorthogonal transform unit 14 and a rate control signal supplied from thelater described rate control unit 18. The quantization unit 15 quantizesthe coefficient data, and outputs the quantized data to the losslesscoding unit 16 and the inverse quantization unit 21. Based on the ratecontrol signal supplied from the rate control unit 18, the quantizationunit 15 switches quantization parameters (quantization scales), tochange the bit rate of the quantized data.

The lossless coding unit 16 receives the quantized data output from thequantization unit 15, and encoding parameter information supplied fromthe later described intra prediction unit 31, the motionprediction/compensation unit 32, and the predicted image/optimum modeselect unit 33. The encoding parameter information contains informationindicating whether the prediction is an intra prediction or an interprediction, macroblock information indicating the macroblock size,information about intra predictions, information about interpredictions, and the like. The lossless coding unit 16 performs alossless coding operation on the quantized data through variable-lengthcoding or arithmetic coding or the like, to generate and output anencoded bit stream to the accumulation buffer 17. The lossless codingunit 16 also performs lossless coding on the encoding parameterinformation, and adds the resultant information to the headerinformation in the encoded bit stream, for example. The quantizationunit 15 and the lossless coding unit 16 are equivalent to the dataprocessing unit that processes the data output from the orthogonaltransform unit 14 to generate an encoded bit stream.

The accumulation buffer 17 stores the encoded bit stream supplied fromthe lossless coding unit 16. The accumulation buffer 17 also outputs thestored encoded bit stream in accordance with the transmission rate ofthe transmission path.

The rate control unit 18 monitors the free space in the accumulationbuffer 17. The rate control unit 18 generates a rate control signal inaccordance with the free space, and outputs the rate control signal tothe quantization unit 15. The rate control unit 18 obtains informationabout the free space from the accumulation buffer 17, for example. Whenthe remaining free space is small, the rate control unit 18 lowers thebit rate of the quantized data through the rate control signal. When theremaining free space in the accumulation buffer 17 is sufficientlylarge, the rate control unit 18 increases the bit rate of the quantizeddata through the rate control signal.

The inverse quantization unit 21 inversely quantizes the quantized datasupplied from the quantization unit 15. The inverse quantization unit 21outputs the coefficient data obtained by performing the inversequantization operation to the inverse orthogonal transform unit 22.

The inverse orthogonal transform unit 22 performs an inverse orthogonaltransform operation on the coefficient data supplied from the inversequantization unit 21, and outputs the resultant data to the additionunit 23.

The addition unit 23 adds the data supplied from the inverse orthogonaltransform unit 22 to the predicted image data supplied from predictedimage/optimum mode select unit 33, to generate reference image data. Theaddition unit 23 outputs the reference image data to the deblockingfilter 24 and the intra prediction unit 31.

The deblocking filter 24 performs a filtering operation to reduce blockdistortions that occur at the time of image encoding. The deblockingfilter 24 performs a filtering operation to remove block distortionsfrom the reference image data supplied from the addition unit 23, andoutputs the reference image data subjected to the filtering operation tothe frame memory 27.

The frame memory 27 holds the reference image data that has beensubjected to the filtering operation and is supplied from the deblockingfilter 24.

The intra prediction unit 31 performs intra prediction operations byusing the image data that is output from the picture rearrangementbuffer 12 and is about the images to be encoded, and the reference imagedata supplied from the addition unit 23. The intra prediction unit 31performs an intra prediction operation for each transform block size inorthogonal transforms and for each prediction mode in intra predictions.The intra prediction unit 31 outputs the generated predicted image datato the predicted image/optimum mode select unit 33. The intra predictionunit 31 also generates the encoding parameter information about theintra prediction operations, and outputs the encoding parameterinformation to the lossless coding unit 16 and the predictedimage/optimum mode select unit 33. The intra prediction unit 31incorporates the macroblock sizes, the transform block sizes, thelocations of the transform blocks in the macroblocks, the predictionmode, and the like into the encoding parameter information.

The intra prediction unit 31 also calculates the cost function value ineach of the intra prediction operations, and selects the intraprediction operation having the smallest cost function value as a resultof the calculation, or the optimum intra prediction operation with thehighest encoding efficiency. The intra prediction unit 31 outputs theencoding parameter information and the cost value in the optimum intraprediction operation, and the predicted image data generated through theoptimum intra prediction operation to the predicted image/optimum modeselect unit 33.

The motion prediction/compensation unit 32 performs inter predictionoperations in all the motion-compensated block sizes corresponding tothe macroblocks, to generate and output predicted image data to thepredicted image/optimum mode select unit 33. Using the reference imagedata that has been subjected to the filtering operation and is read fromthe frame memory 27, the motion prediction/compensation unit 32 detectsmotion vectors from the image of each motion-compensated block size inthe images that are read from the picture rearrangement buffer 12 andare about the images to be encoded. Based on the detected motionvectors, the motion prediction/compensation unit 32 further performs amotion compensation operation on the reference image, to generatepredicted image data. The motion prediction/compensation unit 32 alsogenerates encoding parameter information about the inter predictionoperations, such as encoding parameter information indicating themacroblock sizes, the motion-compensated block sizes, the motionvectors, and the like. The motion prediction/compensation unit 32outputs the encoding parameter information to the lossless coding unit16 and the predicted image/optimum mode select unit 33.

The motion prediction/compensation unit 32 also calculates the costfunction value for each motion-compensated block size, and selects theinter prediction operation having the smallest cost function value as aresult of the calculation, or the inter prediction operation with thehighest encoding efficiency. The motion prediction/compensation unit 32outputs the encoding parameter information and the cost value in theoptimum inter prediction operation, and the predicted image datagenerated through the optimum inter prediction operation to thepredicted image/optimum mode select unit 33.

When the intra prediction unit 31 performs an intra prediction operationfor each transform block size and each prediction mode to select theoptimum intra prediction operation, the predicted image/optimum modeselect unit 33 outputs the encoding parameter information to theorthogonal transform unit 14, the lossless coding unit 16, and outputsthe predicted image data to the subtraction unit 13. When the motionprediction/compensation unit 32 performs an inter prediction operationfor each prediction block to select the optimum inter predictionoperation, the predicted image/optimum mode select unit 33 outputs theencoding parameter information to the orthogonal transform unit 14 andthe lossless coding unit 16, and outputs the predicted image data to thesubtraction unit 13. Further, when selecting either the optimum intraprediction operation or the optimum inter prediction operation as theoptimum mode, the predicted image/optimum mode select unit 33 comparesthe cost function value of the optimum intra prediction operation withthe cost function value of the optimum inter prediction operation. Basedon the comparison result, the predicted image/optimum mode select unit33 selects the prediction operation with the smaller cost functionvalue, or the prediction operation with the higher encoding efficiency,as the optimum mode, and outputs the predicted image data generated inthe selected optimum mode to the subtraction unit 13. Also, thepredicted image/optimum mode select unit 33 outputs the encodingparameter information indicating the prediction operation in the optimummode to the orthogonal transform unit 14 and the lossless coding unit16.

<2. Structure of the Orthogonal Transform Unit>

In an intra prediction operation, the pixels in adjacent encoded blocksare used to make a prediction, and the optimum prediction direction isselected from two or more prediction directions. For example, inH.264/AVC, the four modes of prediction mode 0 through prediction mode 3are set as the prediction modes for blocks of 16×16 pixels. Also, thenine prediction modes of prediction mode 0 through prediction mode 8 areset as the prediction modes for blocks of 8×8 pixels. Further, the nineprediction modes of prediction mode 0 through prediction mode 8 are setas the prediction modes for blocks of 4×4 pixels.

FIG. 2 shows the prediction modes for blocks of 4×4 pixels, for example.In the following, each of the prediction modes shown in FIG. 2 isbriefly described. In FIG. 2, the arrows indicate prediction directions.

FIG. 2(A) illustrates the prediction mode 0 (vertical). The predictionmode 0 is a mode for generating predicted values from reference pixels Athrough D that are adjacent in the vertical direction. FIG. 2(B)illustrates the prediction mode 1 (horizontal). The prediction mode 1 isa mode for generating predicted values from reference pixels I through Lthat are adjacent in the horizontal direction as indicated by thearrows. FIG. 2(C) illustrates the prediction mode 2 (DC). The predictionmode 2 is a mode for generating predicted values from the referencepixels A through D and I through L that are adjacent in the verticaldirection and in the horizontal direction of the block among thirteenreference pixels A through M.

FIG. 2(D) illustrates the prediction mode 3 (diagonal down-left). Theprediction mode 3 is a mode for generating predicted values from thereference pixels A through H that are continuous in the horizontaldirection among the thirteen reference pixels A through M. FIG. 2(E)illustrates the prediction mode 4 (diagonal down-right). The predictionmode 4 is a mode for generating predicted values from the referencepixels A through D and I through M that are adjacent to the block amongthe thirteen reference pixels A through M. FIG. 2(F) illustrates theprediction mode 5 (vertical-right). The prediction mode 5 is a mode forgenerating predicted values from the reference pixels A through D and Ithrough M that are adjacent to the block among the thirteen referencepixels A through M.

FIG. 2(G) illustrates the prediction mode 6 (horizontal-down). Theprediction mode 6 is a mode for generating predicted values from thereference pixels A through D and I through M that are adjacent to theblock among the thirteen reference pixels A through M, like theprediction mode 4 and the prediction mode 5. FIG. 2(H) illustrates theprediction mode 7 (vertical-left). The prediction mode 7 is a mode forgenerating predicted values from the four reference pixels A through Dthat are located above and adjacent to the block, and from the fourreference pixels E through G that follow the four reference pixels Athrough D among the thirteen reference pixels A through M. FIG. 2(I)illustrates the prediction mode 8 (horizontal-up). The prediction mode 8is a mode for generating predicted values from the four reference pixelsI through L that are located on the left side of and adjacent to theblock among the thirteen reference pixels A through M.

When predicted values are generated in this manner, there are many caseswhere a pixel closer to a pixel used in the prediction has a smallerdifference (prediction error) from the predicted value among the pixelsin the block. Therefore, in a case where the prediction mode 0(vertical) is selected as the optimum mode as shown in FIG. 3(A), pixelsP0 through P3 have smaller prediction errors than pixels P12 throughP15, for example. In a case where the prediction mode 1 (horizontal) isselected as shown in FIG. 3(B), pixels P0, P4, P8, and P12 have smallerprediction errors than pixels P3, P7, P11, and P15. In a case where theprediction mode 4 (diagonal down-right) is selected as shown in FIG.3(C), the pixel P0 has a smaller prediction error than the pixel P15. Asdescribed above, prediction errors depend on the prediction modes. Asfor the block locations in each macroblock, there are many cases where ablock closer to an adjacent macroblock that has been encoded has asmaller prediction error, and prediction errors also depend on the blocklocations in the macroblock. Therefore, the orthogonal transform unit 14sets an optimum base for each prediction mode and for each location ofthe blocks to be subjected to orthogonal transforms in the macroblock.In this manner, the orthogonal transform unit 14 optimizes theorthogonal transforms on prediction errors.

In orthogonal transforms, a Karhunen-Loeve transform technique(hereinafter referred to as the KL transform technique) is known as theoptimum transform technique to perform transforms in such a manner thattransformed coefficients are uncorrelated, or the highest encodingefficiency is achieved. However, to determine the base of a KLtransform, it is necessary to generate matrixes based on predictionerrors and calculate eigenvectors corresponding to the eigenvalues ofthe generated matrixes. If the image encoding apparatus calculates thebase each time, the amount of calculation in the image encodingapparatus would become larger. If a calculated base is added to anencoded bit stream, the encoding efficiency would become lower.Therefore, an optimum mode for each location of blocks to beorthogonally transformed in each macroblock and for each prediction modeis calculated through learning in advance. Where the calculated basesare used in the image encoding apparatus and an image decodingapparatus, there is no need for the image encoding apparatus and theimage decoding apparatus to calculate bases, and the structures of theimage encoding apparatus and the image decoding apparatus can be madesimpler than those to calculate bases. Furthermore, since there is noneed to transmit the bases, the encoding efficiency can be increased byusing KL transforms. The learning of bases will be described later.

In an intra prediction, where the macroblock is formed with 16×16pixels, the transform block size that is the block size of the image tobe encoded is 16×16 pixels, 8×8 pixels, or 4×4 pixels, for example.Where the macroblock is formed with 8×8 pixels, the transform block sizeis 8×8 pixels or 4×4 pixels, for example. Therefore, the orthogonaltransform unit 14 is designed to be capable of performing KL transformscompatible with the prediction mode in the block size of 16×16 pixels,8×8 pixels, or 4×4 pixels where the macroblock is formed with 16×16pixels as shown in FIG. 4. Also, the orthogonal transform unit 14 isdesigned to be capable of performing KL transforms compatible with theprediction mode in the block size of 8×8 pixels or 4×4 pixels where themacroblock is formed with 8×8 pixels. Further, in a case where there aretransform blocks in a macroblock, the orthogonal transform unit 14performs KL transforms in accordance with each block location loc in themacroblock.

FIG. 5 illustrates an example structure of the orthogonal transform unit14 that uses KL transforms. The orthogonal transform unit 14 includes a16×16 KL transform unit 141, an 8×8 KL transform unit 142, 2×2 KLtransform units 143 and 146, 4×4 KL transform units 144 and 145, a DCTunit 147, and a coefficient select unit 148.

The 16×16 KL transform unit 141 uses the optimum base learned beforehandfor each prediction mode, and performs a KL transform on predictionerror data for each block of 16×16. The resultant coefficients areoutput to the coefficient select unit 148.

The 8×8 KL transform unit 142 uses the optimum base learned beforehandfor each prediction mode and for each block location in the macroblock,and performs a KL transform on prediction error data for each block of8×8. Where the prediction error data is data compatible with the blocksize of 16×16 pixels, each block of 16×16 pixels includes four blocks of8×8 pixels. Therefore, the 8×8 KL transform unit 142 outputs thecoefficient of the lowest frequency component in each block of 8×8pixels (hereinafter referred to as the “lowest-frequency componentcoefficient”) to the 2×2 KL transform unit 143, and outputs the othercoefficients to the coefficient select unit 148. Also, where theprediction error data is data compatible with the block size of 8×8pixels, the 8×8 KL transform unit 142 performs a KL transform on theprediction error data for each block of 8×8 pixels by using the optimumbase learned beforehand for each prediction mode. The 8×8 KL transformunit 142 outputs the coefficients obtained through the KL transforms tothe coefficient select unit 148.

Using the base compatible with the prediction mode among the optimumbases learned beforehand for the respective prediction modes, the 2×2 KLtransform unit 143 performs a KL transform on the coefficient of each2×2 block supplied from the 8×8 KL transform unit 142, and outputs theresultant coefficients to the coefficient select unit 148.

The 4×4 KL transform unit 144 uses the optimum base learned beforehandfor each prediction mode and for each block location in the macroblock,and performs a KL transform on prediction error data for each block of4×4 pixels. Where the prediction error data is data compatible with theblock size of 16×16 pixels, each block of 16×16 pixels includes sixteenblocks of 4×4 pixels. Therefore, the 4×4 KL transform unit 144 outputsthe lowest-frequency component coefficient in each block of 4×4 pixelsto the 4×4 KL transform unit 145, and outputs the other coefficients tothe coefficient select unit 148. Where the prediction error data is datacompatible with the block size of 8×8 pixels, each block of 8×8 pixelsincludes four blocks of 4×4 pixels. Therefore, the 4×4 KL transform unit144 outputs the lowest-frequency component coefficient in each block of4×4 pixels to the 2×2 KL transform unit 146, and outputs the othercoefficients to the coefficient select unit 148.

Using the base compatible with a prediction mode designated by the 4×4KL transform unit 144 among the optimum bases learned beforehand for therespective prediction modes, the 4×4 KL transform unit 145 performs KLtransforms on the lowest-frequency component coefficients of the blocksof 4×4 supplied from the 4×4 KL transform unit 144. The 4×4 KL transformunit 145 outputs the coefficients obtained through the KL transforms tothe coefficient select unit 148.

Using the base compatible with the prediction mode among the optimumbases learned beforehand for the respective prediction modes, the 2×2 KLtransform unit 146 performs KL transforms on the lowest-frequencycomponent coefficients of the 2×2 blocks supplied from the 4×4 KLtransform unit 144. The 2×2 KL transform unit 146 outputs thecoefficients obtained through the KL transforms to the coefficientselect unit 148.

The DCT unit 147 performs discrete cosine transforms on prediction errordata, and outputs the resultant coefficients to the coefficient selectunit 148.

The coefficient select unit 148 selects coefficients in accordance withthe macroblock size and the transform block size, which is the blocksize corresponding to the prediction error data. Where the macroblocksize is 16×16 pixels, the coefficient select unit 148 selects thecoefficients output from the 16×16 KL transform unit 141, thecoefficients output from the 8×8 KL transform unit 142 and the 2×2 KLtransform unit 143, or the coefficients output from the 4×4 KL transformunit 144 and the 4×4 KL transform unit 145, based on the transform blocksize. The coefficient select unit 148 outputs the selected coefficientsto the quantization unit 15.

Where the macroblock size is 8×8 pixels, the coefficient select unit 148selects the coefficients output from the 8×8 KL transform unit 142 orthe coefficients output from the 4×4 KL transform unit 144 and the 2×2KL transform unit 146, based on the transform block size. Thecoefficient select unit 148 outputs the selected coefficients to thequantization unit 15. When the encoding parameter information suppliedfrom the predicted image/optimum mode select unit 33 indicates an interprediction mode, the coefficient select unit 148 outputs thecoefficients output from the DCT unit 147 to the quantization unit 15.

<3. Operations of the Image Encoding Apparatus>

Next, an image encoding operation is described. FIG. 6 is a flowchartshowing an image encoding operation. In step ST11, the A/D conversionunit 11 performs an A/D conversion on an input image signal.

In step ST12, the picture rearrangement buffer 12 performs imagerearrangement. The picture rearrangement buffer 12 stores the image datasupplied from the A/D conversion unit 11, and rearranges the respectivepictures in encoding order, instead of display order.

In step ST13, the subtraction unit 13 generates prediction error data.The subtraction unit 13 generates the prediction error data bycalculating the differences between the image data of the imagesrearranged in step ST12 and predicted image data selected by thepredicted image/optimum mode select unit 33. The prediction error datahas a smaller data amount than the original image data. Accordingly, thedata amount can be made smaller than in a case where images are directlyencoded.

In step ST14, the orthogonal transform unit 14 performs an orthogonaltransform operation. The orthogonal transform unit 14 orthogonallytransforms the prediction error data supplied from the subtraction unit13. The orthogonal transform unit 14 performs orthogonal transforms suchas Karhunen-Loeve transforms or discrete cosine transforms on theprediction error data, and outputs coefficient data. Operations of theorthogonal transform unit 14 will be described later in detail.

In step ST15, the quantization unit 15 performs a quantizationoperation. The quantization unit 15 quantizes the coefficient data. Inthe quantization, rate control is performed as will be described laterin the description of step ST26.

In step ST16, the inverse quantization unit 21 performs an inversequantization operation. The inverse quantization unit 21 inverselyquantizes the coefficient data quantized by the quantization unit 15,having characteristics compatible with the characteristics of thequantization unit 15.

In step ST17, the inverse orthogonal transform unit 22 performs aninverse orthogonal transform operation. The inverse orthogonal transformunit 22 performs an inverse orthogonal transform on the coefficient datainversely quantized by the inverse quantization unit 21, having thecharacteristics compatible with the characteristics of the orthogonaltransform unit 14.

In step ST18, the addition unit 23 generates reference image data. Theaddition unit 23 generates the reference image data by adding thepredicted image data supplied from the predicted image/optimum modeselect unit 33 to the data of the block location that corresponds to thepredicted image data and has been subjected to the inverse orthogonaltransform.

In step ST19, the deblocking filter 24 performs a filtering operation.The deblocking filter 24 removes block distortions by filtering thereference image data output from the addition unit 23.

In step ST20, the frame memory 27 stores the reference image data. Theframe memory 27 stores the filtered reference image data.

In step ST21, the intra prediction unit 31 and the motionprediction/compensation unit 32 each perform a prediction operation.Specifically, the intra prediction unit 31 performs intra predictionoperations in intra prediction modes, and the motionprediction/compensation unit 32 performs motion prediction/compensationoperations in inter prediction modes. The prediction operations will bedescribed later with reference to FIG. 7. In this step, predictionoperations are performed in all candidate prediction modes, and costfunction values are calculated in all the candidate prediction modes.Based on the calculated cost function values, an optimum intraprediction operation and an optimum inter prediction operation areselected, and the predicted image data generated through the selectedpredicting operations, the cost functions, and the encoding parameterinformation are supplied to the predicted image/optimum mode select unit33.

In step ST22, the predicted image/optimum mode select unit 33 selectspredicted image data. Based on the respective cost function valuesoutput from the intra prediction unit 31 and the motionprediction/compensation unit 32, the predicted image/optimum mode selectunit 33 determines the optimum mode to optimize the encoding efficiency.The predicted image/optimum mode select unit 33 also selects thepredicted image data in the determined optimum mode, and supplies theselected predicted image data to the subtraction unit 13 and theaddition unit 23. This predicted image data is used in the operations insteps ST13 and ST18, as described above.

In step ST23, the predicted image/optimum mode select unit 33 performsan encoding parameter information generating operation. The predictedimage/optimum mode select unit 33 outputs the encoding parameterinformation about the selected predicted image data as the encodingparameter information about the optimum mode to the orthogonal transformunit 14 and the lossless coding unit 16.

In step ST24, the lossless coding unit 16 performs a lossless codingoperation. The lossless coding unit 16 performs lossless coding on thequantized data output from the quantization unit 15. That is, losslesscoding such as variable-length coding or arithmetic coding is performedon the quantized data, to compress the data. At this point, the losslesscoding is also performed on the encoding parameter information suppliedto the lossless coding unit 16 in step ST23 as described above. Further,lossless coding data such as the encoding parameter information is addedto the header information in the encoded bit stream generated byperforming the lossless coding on the quantized data.

In step ST25, the accumulation buffer 17 performs an accumulationoperation. The accumulation buffer 17 stores the encoded bit streamoutput from the lossless coding unit 16. The encoded bit stream storedin the accumulation buffer 17 is read and transmitted to the decodingside via a transmission path where necessary.

In step ST26, the rate control unit 18 performs rate control. The ratecontrol unit 18 controls the quantization operation rate of thequantization unit 15 so that an overflow or an underflow does not occurin the accumulation buffer 17 when the accumulation buffer 17 stores anencoded bit stream.

Referring now to the flowchart of FIG. 7, the prediction operations instep ST21 of FIG. 6 are described.

In step ST31, the intra prediction unit 31 performs intra predictionoperations. The intra prediction unit 31 performs intra predictionoperations on the image of the current block in all the candidateprediction modes. In the intra prediction operations, the referenceimage data supplied from the addition unit 23 is used. In the intrapredictions, intra prediction operations are performed in the respectiveprediction modes as will be described later, and the cost functionvalues in the respective prediction modes are calculated. Based on thecalculated cost function values, the intra prediction operation with thehighest encoding efficiency is selected.

In step ST32, the motion prediction/compensation unit 32 makes interpredictions. Using the filtered reference image data stored in the framememory 27, the motion prediction/compensation unit 32 performs interprediction operations in respective motion-compensated block sizes. Inthe inter predictions, inter prediction operations are performed in therespective motion-compensated block sizes, and the cost function valuesin the respective prediction blocks. Based on the calculated costfunction values, the inter prediction operation with the highestencoding efficiency is selected.

Referring now to the flowchart of FIG. 8, the intra predictionoperations in step ST31 of FIG. 7 are described.

In step ST41, the intra prediction unit 31 provisionally performs intraprediction operations in the respective prediction modes and transformblock sizes. The intra prediction unit 31 provisionally performs anoperation that starts with the generation of predicted image data andthe generation of prediction error data and ends with the losslesscoding in each prediction mode and each transform block size, using thereference image data supplied from the addition unit 23. In each intraprediction operation, the intra prediction unit 31 outputs the encodingparameter information about the intra prediction operation to theorthogonal transform unit 14 and the lossless coding unit 16.

In step ST42, the intra prediction unit 31 calculates the cost functionvalue in each prediction mode and each transform block size. Asspecified in the JM (Joint Model), which is the reference software inH.264/AVC, the cost function value is calculated by the technique ofHigh Complexity Mode or Low Complexity Mode.

Specifically, in the High Complexity Mode, the operation that ends withthe lossless coding operation is provisionally performed as theoperation of the step ST41 in each prediction mode and each transformblock size, to calculate the cost function value expressed by thefollowing equation (1) in each prediction mode and each transform blocksize:Cost(ModeεΩ)=D+λ●R  (1)

Here, Ω represents the universal set of the candidate prediction modesand transform block sizes for encoding the block or the macroblock. Drepresents the energy difference (distortion) between the referenceimage and an input image in a case where encoding is performed in aprediction mode and a transform block size. R represents generatedencoding amount including orthogonal transform coefficients and encodingparameter information, and λ represents the Lagrange multiplier given asthe function of a quantization parameter QP.

That is, to perform encoding in the High Complexity Mode, a provisionalencoding operation needs to be performed in all the candidate predictionmodes and transform block sizes to calculate the above parameters D andR, and therefore, a larger amount of calculation is required.

In the Low Complexity Mode, on the other hand, the generation of apredicted image and the calculation of a header bit such as encodingparameter information are performed as the operation of step ST41 in allthe candidate prediction modes and transform block sizes, and the costfunction value expressed by the following equation (2) is calculated ineach prediction mode:Cost(ModeεΩ)=D+QPtoQuant(QP)●Header_Bit  (2)

Here, Ω represents the universal set of the candidate prediction modesand transform block sizes for encoding the block or the macroblock. Drepresents the energy difference (distortion) between the referenceimage and an input image in a case where encoding is performed in aprediction mode and a transform block size. Header_Bit represents theheader bit corresponding to the prediction mode and the transform blocksize, and QPtoQuant is the function given as the function of thequantization parameter QP.

That is, in the Low Complexity Mode, a prediction operation needs to beperformed in each prediction mode and each transform block size, but anydecoded image is not required. Accordingly, the amount of calculationcan be smaller than that required in the High Complexity Mode.

In step ST43, the intra prediction unit 31 determines the optimum intraprediction operation. Based on the cost function values calculated instep ST42, the intra prediction unit 31 selects the one intra predictionoperation with the smallest cost function value among the calculatedcost function values, and determines the selected intra predictionoperation to be the optimum intra prediction operation.

Referring now to the flowchart of FIG. 9, the inter predictionoperations in step ST32 of FIG. 7 are described.

In step ST51, the motion prediction/compensation unit 32 provisionallyperforms an inter prediction operation in each motion-compensated blocksize. Using the image data of the current block to be encoded and thereference image data, the motion prediction/compensation unit 32provisionally makes a motion prediction in each motion-compensated blocksize. Based on the detected motion vectors, the motionprediction/compensation unit 32 performs motion compensation on thereference image data, to generate predicted image data and the like. Ineach inter prediction operation, the motion prediction/compensation unit32 outputs the encoding parameter information about the inter predictionoperation to the orthogonal transform unit 14 and the lossless codingunit 16.

In step ST52, the motion prediction/compensation unit 32 calculates thecost function value for each motion-compensated block size. Using theabove mentioned equation (1) or (2), the motion prediction/compensationunit 32 calculates the cost function values. In the calculation of thecost function values, generated encoding amount including encodingparameter information and the like is used. The cost function valuecalculation in the inter prediction modes involves the evaluations ofcost function values in Skip Mode and Direct Mode specified inH.264/AVC.

In step ST53, the motion prediction/compensation unit 32 determines theoptimum inter prediction operation. Based on the cost function valuescalculated in step ST52, the motion prediction/compensation unit 32selects the one inter prediction operation with the smallest costfunction value among the calculated cost function values, and determinesthe selected inter prediction operation to be the optimum interprediction operation.

Referring now to the flowchart in FIG. 10, the encoding parameterinformation generating operation in step ST23 of FIG. 6 in the case ofan intra prediction operation is described. Encoding parameterinformation is generated by the intra prediction unit 31, as describedabove. Where the predicted image/optimum mode select unit 33 selects theoptimum mode, the encoding parameter information corresponding to theselected prediction operation may be generated by the predictedimage/optimum mode select unit 33.

In step ST61, the intra prediction unit 31 determines whether themacroblock size is 16×16 pixels. If the macroblock size is 16×16 pixels,the intra prediction unit 31 moves on to step ST62, and if not 16×16pixels, moves on to step ST63.

In step ST62, the intra prediction unit 31 sets transform block sizeinformation in 16×16 pixels, and moves on to step ST65. Where thetransform block size for the orthogonal transform unit 14 to perform KLtransforms is 4×4 pixels, the intra prediction unit 31 sets thetransform block size information indicating the transform block size to“0”. Where the transform block size for the orthogonal transform unit 14to perform KL transforms is 8×8 pixels, the intra prediction unit 31sets the transform block size information to “1”. Where the transformblock size is 16×16 pixels, the intra prediction unit 31 sets thetransform block size information to “2”.

In step ST63, the intra prediction unit 31 determines whether themacroblock size is 8×8 pixels. If the macroblock size is 8×8 pixels, theintra prediction unit 31 moves on to step ST64, and if not 8×8 pixels,moves on to step ST65.

In step ST64, the intra prediction unit 31 sets transform block sizeinformation in 8×8 pixels, and moves on to step ST65. Where thetransform block size for the orthogonal transform unit 14 to perform KLtransforms is 4×4 pixels, the intra prediction unit 31 sets thetransform block size information to “0”. Where the transform block sizefor the orthogonal transform unit 14 to perform KL transforms is 8×8pixels, the intra prediction unit 31 sets the transform block sizeinformation to “1”.

In step ST65, the intra prediction unit 31 generates the encodingparameter information. The intra prediction unit 31 forms the encodingparameter information, using the information indicating that theprediction is an intra prediction, the macroblock size, the transformblock size information, the prediction mode, the block location in themacroblock, and the like.

Referring now to the flowchart in FIG. 11, the orthogonal transformoperation is described. In step ST71, the orthogonal transform unit 14determines whether there are intra predictions. If the encodingparameter information indicates intra predictions, the orthogonaltransform unit 14 moves on to step ST72, and if not intra predictions,moves on to step ST81.

In step ST72, the orthogonal transform unit 14 determines whether themacroblock size is 16×16 pixels. If the encoding parameter informationindicates that the macroblock size is 16×16 pixels, the orthogonaltransform unit 14 moves on to step ST73, and if not 16×16 pixels, moveson to step ST78.

In step ST73, the orthogonal transform unit 14 determines whether thetransform block size is 4×4 pixels. If the encoding parameterinformation indicates that the transform block size is 4×4 pixels, theorthogonal transform unit 14 moves on to step ST74, and if not 4×4pixels, moves on to step ST75.

In step ST74, the orthogonal transform unit 14 performs a 4×4 orthogonaltransform operation. Using the bases learned beforehand in accordancewith the prediction mode and the block locations, the orthogonaltransform unit 14 performs a KL transform on each block of 4×4 pixels.Since a block of 16×16 pixels includes sixteen blocks of 4×4 pixels, aKL transform is performed sixteen times herein. Further, the orthogonaltransform unit 14 selects the lowest-frequency component coefficientsfrom the coefficients obtained by performing the KL transforms on theblocks of 4×4 pixels, and performs KL transforms on the selected 4×4coefficients by using the base compatible with the prediction mode. Theorthogonal transform unit 14 outputs the coefficients obtained byperforming the KL transforms on the lowest-frequency componentcoefficients, and the coefficients other than the lowest-frequencycomponent coefficients, to the quantization unit 15. That is, thecoefficient select unit 148 of the orthogonal transform unit 14 shown inFIG. 5 selects the coefficients output from the 4×4 KL transform units144 and 146, and outputs the selected coefficients to the quantizationunit 15.

In step ST75, the orthogonal transform unit 14 determines whether thetransform block size is 8×8 pixels. If the encoding parameterinformation indicates that the transform block size is 8×8 pixels, theorthogonal transform unit 14 moves on to step ST76, and if not 8×8pixels, moves on to step ST77.

In step ST76, the orthogonal transform unit 14 performs an 8×8orthogonal transform operation. Using the bases learned beforehand inaccordance with the prediction mode and the block locations, theorthogonal transform unit 14 performs a KL transform on each block of8×8 pixels. Since a block of 16×16 pixels includes four blocks of 8×8pixels, a KL transform is performed four times herein. Further, theorthogonal transform unit 14 selects the lowest-frequency componentcoefficients from the coefficients obtained by performing the KLtransforms on the blocks of 8×8 pixels, and performs KL transforms onthe selected 2×2 coefficients by using the base compatible with theprediction mode. The orthogonal transform unit 14 outputs thecoefficients obtained by performing the KL transforms on thelowest-frequency component coefficients, and the coefficients other thanthe lowest-frequency component coefficients, to the quantization unit15. That is, the coefficient select unit 148 of the orthogonal transformunit 14 shown in FIG. 5 selects the coefficients output from the 8×8 KLtransform unit 142 and the 2×2 KL transform unit 143, and outputs theselected coefficients to the quantization unit 15.

In step ST77, the orthogonal transform unit 14 performs a 16×16orthogonal transform operation. Using the base learned beforehand inaccordance with the prediction mode, the orthogonal transform unit 14performs a KL transform on the block of 16×16 pixels, and outputs theresultant coefficients to the quantization unit 15. That is, thecoefficient select unit 148 of the orthogonal transform unit 14 shown inFIG. 5 selects the coefficients output from the 16×16 KL transform unit141, and outputs the selected coefficients to the quantization unit 15.

When moving from step ST72 on to step ST78, the orthogonal transformunit 14 determines whether the transform block size is 4×4 pixels. Ifthe encoding parameter information indicates that the transform blocksize is 4×4 pixels, the orthogonal transform unit 14 moves on to stepST79, and if not 4×4 pixels, moves on to step ST80.

In step ST79, the orthogonal transform unit 14 performs a 4×4 orthogonaltransform operation. Using the bases learned beforehand in accordancewith the prediction mode and the block locations, the orthogonaltransform unit 14 performs a KL transform on each block of 4×4 pixels.Since a block of 8×8 pixels includes four blocks of 4×4 pixels, a KLtransform is performed four times herein. Further, the lowest-frequencycomponent coefficients are selected from the coefficients obtainedthrough the KL transforms performed on the blocks of 4×4 pixels, and KLtransforms are performed on the selected 2×2 coefficients by using thebase compatible with the prediction mode. The orthogonal transform unit14 outputs the coefficients obtained by performing the KL transforms onthe lowest-frequency component coefficients, and the coefficients otherthan the lowest-frequency component coefficients, to the quantizationunit 15. That is, the coefficient select unit 148 of the orthogonaltransform unit 14 shown in FIG. 5 selects the coefficients output fromthe 4×4 KL transform units 144 and the 2×2 KL transform unit 146, andoutputs the selected coefficients to the quantization unit 15.

In step ST80, the orthogonal transform unit 14 performs an orthogonaltransform on each block of 8×8 pixels. Using the base learned beforehandin accordance with the prediction mode, the orthogonal transform unit 14performs a KL transform on the block of 8×8 pixels, and outputs theresultant coefficients to the quantization unit 15. That is, thecoefficient select unit 148 of the orthogonal transform unit 14 shown inFIG. 5 selects the coefficients output from the 8×8 KL transform unit142, and outputs the selected coefficients to the quantization unit 15.

In step ST81, the orthogonal transform unit 14 performs discrete cosinetransforms (DCT). The orthogonal transform unit 14 outputs thecoefficients obtained through the discrete cosine transforms to thequantization unit 15. That is, the coefficient select unit 148 of theorthogonal transform unit 14 shown in FIG. 5 selects the coefficientsoutput from the DCT unit 147, and outputs the selected coefficients tothe quantization unit 15.

FIG. 12 are diagrams for explaining the orthogonal transform operation.Where the macroblock size is 16×16 pixels as shown in FIG. 12(A) and thetransform block size is 4×4 pixels, sixteen transform blocks arecontained in the macroblock as shown in FIG. 12(B). It should be notedthat the numbers in the blocks represent the block locations loc. Usingthe base optimized for the prediction mode and block location of eachblock, the 4×4 KL transform unit 144 of the orthogonal transform unit 14performs a KL transform on each transform block, to generate thecoefficients of the respective blocks as shown in FIG. 12(C). Further,using the lowest-frequency component coefficients (indicated by theshaded portions) in the respective blocks, the 4×4 KL transform unit 145forms a 4×4 block as shown in FIG. 12(D). Using the base optimized inaccordance with the prediction mode, the 4×4 KL transform unit 145performs a KL transform on this block, and generates the coefficient ofeach block as shown in FIG. 12(E). The orthogonal transform unit 14outputs the coefficients shown in FIG. 12(E), and the coefficients otherthan the lowest-frequency component coefficients in FIG. 12(C), to thequantization unit 15.

Where the macroblock size is 8×8 pixels as shown in FIG. 12(F) and thetransform block size is 4×4 pixels, four transform blocks are containedin the macroblock as shown in FIG. 12(G). It should be noted that thenumbers in the blocks represent the block locations loc. Using the baseoptimized for the prediction mode and block location of each block, the4×4 KL transform unit 144 of the orthogonal transform unit 14 performs aKL transform on each transform block, to generate the coefficients ofthe respective blocks as shown in FIG. 12(H). Further, using thelowest-frequency component coefficients (indicated by the shadedportions) in the respective blocks, the 2×2 KL transform unit 146 formsa 2×2 block as shown in FIG. 12(I). Using the base optimized inaccordance with the prediction mode, the 2×2 KL transform unit 146performs a KL transform on this block, and generates the coefficient ofeach block as shown in FIG. 12(J). The orthogonal transform unit 14outputs the coefficients shown in FIG. 12(J), and the coefficients otherthan the lowest-frequency component coefficients in FIG. 12(H), to thequantization unit 15.

As described above, with the image encoding apparatus and method of thepresent invention, the bases that are set beforehand in accordance withthe block locations of the transform blocks in a macroblock are used toperform orthogonal transforms at the time of image data encoding.Accordingly, transforms optimized in accordance with the block locationscan be performed, and encoding efficiency can be increased. Also, thebases that are set beforehand in accordance not only with the blocklocations but also with the prediction modes are used to performorthogonal transforms that are optimized to a greater extent.Accordingly, the encoding efficiency can be further increased. As theencoding efficiency is increased, the image quality can be improvedwithout an increase in the encoded bit stream data amount, for example.

<4. Structure of an Image Decoding Apparatus>

An encoded bit stream generated by encoding an input image is suppliedto an image decoding apparatus via a predetermined transmission path ora recording medium or the like, and is decoded therein.

FIG. 13 illustrates the structure of the image decoding apparatus. Theimage decoding apparatus 50 includes an accumulation buffer 51, alossless decoding unit 52, an inverse quantization unit 53, an inverseorthogonal transform unit 54, an addition unit 55, a deblocking filter56, a picture rearrangement buffer 57, and a digital/analog conversionunit (D/A conversion unit) 58. The image decoding apparatus 50 furtherincludes a frame memory 61 an intra prediction unit 62, a motioncompensation unit 63, and a selector 64.

The accumulation buffer 51 stores a transmitted encoded bit stream. Thelossless decoding unit 52 decodes the encoded bit stream supplied fromthe accumulation buffer 51 by a technique compatible with the encodingtechnique used by the lossless coding unit 16 of FIG. 1.

The lossless decoding unit 52 decodes the header information in theencoded bit stream, and outputs the resultant encoding parameterinformation to the intra prediction unit 62, the motion compensationunit 63, and the deblocking filter 56. Using the motion vectors of theblock to be decoded and decoded adjacent blocks, the lossless decodingunit 52 sets predicted motion vector candidates. Based on the predictedmotion vector select information obtained by performing losslessdecoding on the encoded bit stream, the lossless decoding unit 52selects motion vectors from the predicted motion vector candidates, andsets the selected motion vectors as predicted motion vectors. Thelossless decoding unit 52 adds the predicted motion vectors to motionvector differences obtained by performing lossless decoding on theencoded bit stream, and calculates and outputs the motion vectors of theblock to be decoded to the motion compensation unit 63.

The inverse quantization unit 53 inversely quantizes the quantized datadecoded by the lossless decoding unit 52, using a technique compatiblewith the quantization technique used by the quantization unit 15 ofFIG. 1. The inverse orthogonal transform unit 54 performs an inverseorthogonal transform on the output from the inverse quantization unit 53by a technique compatible with the orthogonal transform technique usedby the orthogonal transform unit 14 of FIG. 1, and outputs the result tothe addition unit 55.

The addition unit 55 generates decoded image data by adding the datasubjected to the inverse orthogonal transform to predicted image datasupplied from the selector 64, and outputs the decoded image data to thedeblocking filter 56 and the intra prediction unit 62.

The deblocking filter 56 performs filtering on the decoded image datasupplied from the addition unit 55, and removes block distortions. Theresultant data is supplied to and stored into the frame memory 61, andis also output to the picture rearrangement buffer 57.

The picture rearrangement buffer 57 performs image rearrangement.Specifically, the frame order rearranged in the order of encodingperformed by the picture rearrangement buffer 12 of FIG. 1 is rearrangedin the original display order, and is output to the D/A conversion unit58.

The D/A conversion unit 58 performs a D/A conversion on the image datasupplied from the picture rearrangement buffer 57, and outputs theconverted image data to a display (not shown) to display the images.

The frame memory 61 holds the decoded image data that has been subjectedto the filtering operation and is supplied from the deblocking filter24.

Based on the encoding parameter information supplied from the losslessdecoding unit 52, the intra prediction unit 62 generates predictedimages, and outputs the generated predicted image data to the selector64.

Based on the encoding parameter information and motion vectors suppliedfrom the lossless decoding unit 52, the motion compensation unit 63performs motion compensation, to generate and output predicted imagedata to the selector 64. Specifically, based on the motion vectorssupplied from the lossless decoding unit 52 and reference frameinformation, the motion compensation unit 63 performs motioncompensation using the motion vectors on the reference image indicatedby the reference frame information, and generates predicted image dataof a motion-compensated block size.

The selector 64 supplies the predicted image data generated by the intraprediction unit 62 to the addition unit 55. The selector 64 alsosupplies the predicted image data generated by the motion compensationunit 63 to the addition unit 55.

<5. Structure of the Inverse Orthogonal Transform Unit>

FIG. 14 illustrates the structure of the inverse orthogonal transformunit 54. The inverse orthogonal transform unit 54 includes a 16×16 KLinverse transform unit 541, 2×2 KL inverse transform units 542 and 545,an 8×8 KL inverse transform unit 543, 4×4 KL inverse transform units 544and 546, an IDCT unit 547, and a data select unit 548.

The 16×16 KL inverse transform unit 541 performs KL inverse transformscorresponding to the KL transforms performed by the 16×16 KL transformunit 141 shown in FIG. 5. Using the base compatible with the predictionmode (optimum prediction mode) indicated by the encoding parameterinformation about the optimum mode supplied from the lossless decodingunit 52, the 16×16 KL inverse transform unit 541 performs KL inversetransforms on inversely quantized data output from the inversequantization unit 53. The 16×16 KL inverse transform unit 541 outputsthe image data obtained by performing the KL inverse transforms to thedata select unit 548.

The 2×2 KL inverse transform unit 542 performs KL inverse transformscorresponding to the KL transforms performed by the 2×2 KL transformunit 143 shown in FIG. 5. Using the base compatible with the predictionmode indicated by the optimum mode encoding parameter information, the2×2 KL inverse transform unit 542 performs KL inverse transforms on theinversely quantized data output from the inverse quantization unit 53.The 2×2 KL inverse transform unit 542 outputs the lowest-frequencycomponent coefficients obtained by performing the KL inverse transformsto the 8×8 KL inverse transform unit 543.

The 8×8 KL inverse transform unit 543 performs KL inverse transformscorresponding to the KL transforms performed by the 8×8 KL transformunit 143 shown in FIG. 5. Based on the optimum mode encoding parameterinformation supplied from the lossless decoding unit 52, the 8×8 KLinverse transform unit 543 performs KL inverse transforms. Where themacroblock size is 16×16 pixels, for example, the 8×8 KL inversetransform unit 543 performs KL inverse transforms on thelowest-frequency component coefficients output from the 2×2 KL inversetransform unit 542 and on the inversely quantized data output from theinverse quantization unit 53, using the bases compatible with theprediction mode indicated by the optimum mode encoding parameterinformation and the block locations. The 8×8 KL inverse transform unit543 outputs the image data obtained by performing the KL inversetransforms to the data select unit 548. Where the macroblock size is 8×8pixels, the 8×8 KL inverse transform unit 543 performs KL inversetransforms on the inversely quantized data output from the inversequantization unit 53, using the bases compatible with the predictionmode and the block locations. The resultant image data is output to thedata select unit 548.

The 4×4 KL inverse transform unit 544 performs KL inverse transformscorresponding to the KL transforms performed by the 4×4 KL transformunit 145 shown in FIG. 5. Using the base compatible with the predictionmode indicated by the optimum mode encoding parameter information, the4×4 KL inverse transform unit 544 performs KL inverse transforms on theinversely quantized data output from the inverse quantization unit 53.The 4×4 KL inverse transform unit 544 outputs the lowest-frequencycomponent coefficients obtained by performing the KL inverse transformsto the 4×4 KL inverse transform unit 546.

The 2×2 KL inverse transform unit 545 performs KL inverse transformscorresponding to the KL transforms performed by the 2×2 KL transformunit 146 shown in FIG. 5. Using the base compatible with the predictionmode indicated by the optimum mode encoding parameter information, the2×2 KL inverse transform unit 545 performs KL inverse transforms on theinversely quantized data output from the inverse quantization unit 53.The 2×2 KL inverse transform unit 545 outputs the lowest-frequencycomponent coefficients obtained by performing the KL inverse transformsto the 4×4 KL inverse transform unit 546.

The 4×4 KL inverse transform unit 546 performs KL inverse transformscorresponding to the KL transforms performed by the 4×4 KL transformunit 144 shown in FIG. 5. Based on the optimum mode encoding parameterinformation supplied from the lossless decoding unit 52, the 4×4 KLinverse transform unit 546 performs KL inverse transforms. Where themacroblock size is 16×16 pixels, for example, the 4×4 KL inversetransform unit 546 performs KL inverse transforms on thelowest-frequency component coefficients output from the 4×4 KL inversetransform unit 544 and on the inversely quantized data output from theinverse quantization unit 53, using the bases compatible with theprediction mode indicated by the optimum mode encoding parameterinformation and the block locations. The 4×4 KL inverse transform unit546 outputs the image data obtained by performing the KL inversetransforms to the data select unit 548. Where the macroblock size is 8×8pixels, the 4×4 KL inverse transform unit 546 performs KL inversetransforms on the lowest-frequency component coefficients output fromthe 2×2 KL inverse transform unit 545 and on the inversely quantizeddata output from the inverse quantization unit 53, using the basescompatible with the prediction mode and the block locations. The 4×4 KLinverse transform unit 546 outputs the image data obtained by performingthe KL inverse transforms to the data select unit 548.

Using the inversely quantized data output from the inverse quantizationunit 53, the IDCT unit 547 performs inverse discrete cosine transforms,and outputs the resultant image data to the data select unit 548.

Based on the encoding parameter information, the data select unit 548selects image data from the image data output from the 16×16 KL inversetransform unit 541, the 8×8 KL inverse transform unit 543, the 4×4 KLinverse transform unit 546, and the IDCT unit 547. The data select unit548 outputs the selected image data as prediction error data to theaddition unit 55.

<6. Operations of the Image Decoding Apparatus>

Referring now to the flowchart in FIG. 15, an image decoding operationto be performed by the image decoding apparatus 50 is described.

In step ST91, the accumulation buffer 51 stores a transmitted encodedbit stream. In step ST92, the lossless decoding unit 52 performs alossless decoding operation. The lossless decoding unit 52 decodes theencoded bit stream supplied from the accumulation buffer 51.Specifically, the quantized data of each picture encoded by the losslesscoding unit 16 of FIG. 1 is obtained. The lossless decoding unit 52performs lossless decoding on the encoding parameter informationcontained in the header information in the encoded bit stream, andsupplies the resultant encoding parameter information to the deblockingfilter 56 and the selector 64. Further, in a case where the encodingparameter information is information about an intra prediction mode, thelossless decoding unit 52 outputs the encoding parameter information tothe intra prediction unit 62. In a case where the encoding parameterinformation is information about an inter prediction mode, the losslessdecoding unit 52 outputs the encoding parameter information to themotion compensation unit 63.

In step ST93, the inverse quantization unit 53 performs an inversequantization operation. The inverse quantization unit 53 inverselyquantizes the quantized data decoded by the lossless decoding unit 52,having characteristics compatible with the characteristics of thequantization unit 15 of FIG. 1.

In step ST94, the inverse orthogonal transform unit 54 performs aninverse orthogonal transform operation. The inverse orthogonal transformunit 54 performs inverse orthogonal transforms compatible with theorthogonal transforms performed by the orthogonal transform unit 14 ofFIG. 1, on the inversely quantized data supplied from the inversequantization unit 53.

In step ST95, the addition unit 55 generates decoded image data. Theaddition unit 55 adds the prediction error data obtained through theinverse orthogonal transform operation to predicted image data selectedin step ST99, which will be described later, and generates the decodedimage data. In this manner, the original images are decoded.

In step ST96, the deblocking filter 56 performs a filtering operation.The deblocking filter 56 performs filtering on the decoded image dataoutput from the addition unit 55, and removes block distortionscontained in the decoded images.

In step ST97, the frame memory 61 performs a decoded image data storingoperation.

In step ST98, the intra prediction unit 62 and the motion compensationunit 63 performs prediction operations. The intra prediction unit 62 andthe motion compensation unit 63 each perform prediction operations inaccordance with the encoding parameter information supplied from thelossless decoding unit 52.

Specifically, in a case where the encoding parameter informationsupplied from the lossless decoding unit 52 indicates intra predictions,the intra prediction unit 62 performs an intra prediction operationbased on the encoding parameter information, to generate predicted imagedata. In a case where the encoding parameter information supplied fromthe lossless decoding unit 52 indicates inter predictions, the motioncompensation unit 63 performs motion compensation based on the encodingparameter information, to generate predicted image data.

In step ST99, the selector 64 selects predicted image data.Specifically, the selector 64 selects predicted image data supplied fromthe intra prediction unit 62 and predicted image data generated by themotion compensation unit 63, and supplies the selected predicted imagedata to the addition unit 55, which adds the selected predicted imagedata to the output from the inverse orthogonal transform unit 54 in stepST95, as described above.

In step ST100, the picture rearrangement buffer 57 performs imagerearrangement. Specifically, the order of frames rearranged for encodingby the picture rearrangement buffer 12 of the image encoding apparatus10 of FIG. 1 is rearranged in the original display order by the picturerearrangement buffer 57.

In step ST101, the D/A conversion unit 58 performs a D/A conversion onthe image data supplied from the picture rearrangement buffer 57. Theimages are output to the display (not shown), and are displayed.

Referring now to the flowchart shown in FIG. 16, the inverse orthogonaltransform operation is described. In step ST111, the inverse orthogonaltransform unit 54 determines whether there are intra predictions. Theinverse orthogonal transform unit 54 determines whether the blocks to bedecoded are intra predictions, based on the encoding parameterinformation extracted from the encoded bit stream by the losslessdecoding unit 52, for example. If the encoding parameter informationindicates intra predictions, the inverse orthogonal transform unit 54moves on to step ST112, and if not intra predictions or if interpredictions, moves on to step ST121.

In step ST112, the inverse orthogonal transform unit 54 determineswhether the macroblock size is 16×16 pixels. If the macroblock sizeindicated by the encoding parameter information is 16×16 pixels, theinverse orthogonal transform unit 54 moves on to step ST113, and if not16×16 pixels, moves on to step ST118.

In step ST113, the inverse orthogonal transform unit 54 determineswhether the transform block size is 4×4 pixels. If the transform blocksize information in the encoding parameter information is “0” and thetransform block size is 4×4 pixels, the inverse orthogonal transformunit 54 moves on to step ST114, and if not “0”, moves on to step ST115.

In step ST114, the inverse orthogonal transform unit 54 performs a 4×4inverse orthogonal transform operation. Using the bases learnedbeforehand in accordance with the prediction mode and the blocklocations, the inverse orthogonal transform unit 54 performs 4×4 KLinverse transforms. Where the macroblock size is 16×16 pixels, in theencoding, a KL transform is performed sixteen times, and KL transformsare further performed after the lowest-frequency component coefficientsare selected from the coefficients obtained by performing the KLtransforms. Therefore, the inverse orthogonal transform unit 54 performsKL inverse transforms on the inversely quantized data of thelowest-frequency component coefficients, using the base compatible withthe prediction mode. The inverse orthogonal transform unit 54 alsoperforms KL inverse transforms on sixteen blocks formed with thelowest-frequency component coefficients obtained through the KL inversetransforms and the coefficients of the other components, using the basescompatible with prediction mode and the block locations. The inverseorthogonal transform unit 54 outputs the prediction error data obtainedby performing the KL inverse transforms to the addition unit 55. Thatis, the data select unit 548 of the inverse orthogonal transform unit 54shown in FIG. 14 selects the data obtained by the 4×4 KL inversetransform unit 546 performing KL inverse transforms using the outputfrom the 4×4 KL inverse transform unit 544, and outputs the selecteddata to the addition unit 55.

In step ST115, the inverse orthogonal transform unit 54 determineswhether the transform block size is 8×8 pixels. If the transform blocksize information in the encoding parameter information is “1” and thetransform block size is 8×8 pixels, the inverse orthogonal transformunit 54 moves on to step ST116, and if not “1”, moves on to step ST117.

In step ST116, the inverse orthogonal transform unit 54 performs an 8×8inverse orthogonal transform operation. Using the bases learnedbeforehand in accordance with the prediction mode and the blocklocations, the inverse orthogonal transform unit 54 performs 8×8 KLinverse transforms. Where the macroblock size is 16×16 pixels, in theencoding, a KL transform is performed four times, and KL transforms arefurther performed after the lowest-frequency component coefficients areselected from the coefficients obtained by performing the KL transforms.Therefore, the inverse orthogonal transform unit 54 performs KL inversetransforms on the inversely quantized data of the lowest-frequencycomponent coefficients, using the base compatible with the predictionmode. The inverse orthogonal transform unit 54 also performs KL inversetransforms on four blocks formed with the lowest-frequency componentcoefficients obtained through the KL inverse transforms and thecoefficients of the other components, using the bases compatible withprediction mode and the block locations. The inverse orthogonaltransform unit 54 outputs the prediction error data obtained byperforming the KL inverse transforms to the addition unit 55. That is,the data select unit 548 of the inverse orthogonal transform unit 54shown in FIG. 14 selects the data obtained by the 8×8 KL inversetransform unit 543 performing KL inverse transforms using the outputfrom the 2×2 KL inverse transform unit 542, and outputs the selecteddata to the addition unit 55.

In step ST117, the inverse orthogonal transform unit 54 performs a 16×16inverse orthogonal transform operation. Using the base learnedbeforehand in accordance with the prediction mode, the inverseorthogonal transform unit 54 performs 16×16 KL inverse transforms. Theinverse orthogonal transform unit 54 outputs the prediction error dataobtained by performing the KL inverse transforms to the addition unit55. That is, the data select unit 548 of the inverse orthogonaltransform unit 54 shown in FIG. 14 selects the data obtained by the16×16 KL inverse transform unit 541 performing KL inverse transforms,and outputs the selected data to the addition unit 55.

When moving from step ST112 on to step ST118, the inverse orthogonaltransform unit 54 determines whether the transform block size is 4×4pixels. If the transform block size information in the encodingparameter information is “0” and the transform block size is 4×4 pixels,the inverse orthogonal transform unit 54 moves on to step ST119, and ifnot “0”, moves on to step ST120.

In step ST119, the inverse orthogonal transform unit 54 performs a 4×4inverse orthogonal transform operation. Using the bases learnedbeforehand in accordance with the prediction mode and the blocklocations, the inverse orthogonal transform unit 54 performs the 4×4 KLinverse transform operation. Where the macroblock size is 8×8 pixels, inthe encoding, a KL transform is performed four times, and KL transformsare further performed after the lowest-frequency component coefficientsare selected from the coefficients obtained by performing the KLtransforms. Therefore, the inverse orthogonal transform unit 54 performsKL inverse transforms on the inversely quantized data of thelowest-frequency component coefficients, using the base compatible withthe prediction mode. The inverse orthogonal transform unit 54 alsoperforms KL inverse transforms on four blocks formed with thelowest-frequency component coefficients obtained through the KL inversetransforms and the coefficients of the other components, using the basescompatible with prediction mode and the block locations. The inverseorthogonal transform unit 54 outputs the prediction error data obtainedby performing the KL inverse transforms to the addition unit 55. Thatis, the data select unit 548 of the inverse orthogonal transform unit 54shown in FIG. 14 selects the data obtained by the 4×4 KL inversetransform unit 546 performing KL inverse transforms using the outputfrom the 2×2 KL inverse transform unit 545, and outputs the selecteddata to the addition unit 55.

In step ST120, the inverse orthogonal transform unit 54 performs an 8×8inverse orthogonal transform operation. Using the base learnedbeforehand in accordance with the prediction mode, the inverseorthogonal transform unit 54 performs 8×8 KL inverse transforms. Theinverse orthogonal transform unit 54 outputs the prediction error dataobtained by performing the KL inverse transforms to the addition unit55. That is, the data select unit 548 of the inverse orthogonaltransform unit 54 shown in FIG. 14 selects the data obtained by the 8×8KL inverse transform unit 543 performing KL inverse transforms, andoutputs the selected data to the addition unit 55.

In step ST121, the inverse orthogonal transform unit 54 performs inversediscrete cosine transforms (IDCT). The inverse orthogonal transform unit54 outputs the coefficients obtained by performing the inverse discretecosine transforms to the addition unit 55. That is, the data select unit548 of the inverse orthogonal transform unit 54 shown in FIG. 14 selectsthe data output from the IDCT unit 547, and outputs the selected data tothe addition unit 55.

FIG. 17 are diagrams for explaining the inverse orthogonal transformoperation, and shows example inverse orthogonal transforms of transformcoefficients generated by the orthogonal transform operation of FIG. 12.

There is a case where the macroblock size is 16×16 pixels, and thetransform block size is 4×4 pixels, for example. In this case, the 4×4KL inverse transform unit 544 performs KL inverse transforms onKL-transformed data (inversely quantized data) of the lowest-frequencycomponent coefficients shown in FIG. 17(A), using the base compatiblewith the prediction mode indicated by optimum mode encoding parameterinformation. Through the KL inverse transforms, the 4×4 KL inversetransform unit 544 generates the coefficients of the lowest frequencycomponents shown in FIG. 17(B). The 4×4 KL inverse transform unit 546returns the lowest-frequency component coefficients and the otherKL-transformed data (inversely quantized data) to the coefficients ofthe respective blocks, as shown in FIG. 17(C). Further, the 4×4 KLinverse transform unit 546 performs a KL inverse transform on each ofthe sixteen 4×4 blocks, using the bases compatible with the predictionmode indicated by the encoding parameter information and the blocklocations, as shown in FIG. 17(D). As a result, the prediction errordata shown in FIG. 17(E) is generated. The data select unit 548 selectsthe generated prediction error data, and outputs the selected data tothe addition unit 55.

There is also a case where the macroblock size is 8×8 pixels, and thetransform block size is 4×4 pixels, for example. In this case, the 2×2KL inverse transform unit 545 performs KL inverse transforms onKL-transformed data (inversely quantized data) of the lowest-frequencycomponent coefficients shown in FIG. 17(F), using the base compatiblewith the prediction mode indicated by the optimum mode encodingparameter information. Through the KL inverse transforms, the 2×2 KLinverse transform unit 545 generates the lowest-frequency componentcoefficients shown in FIG. 17(G). The 4×4 KL inverse transform unit 546returns the lowest-frequency component coefficients and the otherKL-transformed data (inversely quantized data) to the coefficients ofthe respective blocks, as shown in FIG. 17(H). Further, the 4×4 KLinverse transform unit 546 performs a KL inverse transform on each ofthe four 4×4 blocks, using the bases compatible with the prediction modeindicated by the encoding parameter information and the block locations,as shown in FIG. 17(I). As a result, the prediction error data shown inFIG. 17(J) is generated. The data select unit 548 selects the generatedprediction error data, and outputs the selected data to the additionunit 55.

Referring now to the flowchart in FIG. 18, the prediction operations ofstep ST98 of FIG. 15 are described.

In step ST131, the lossless decoding unit 52 determines whether thecurrent blocks are intra-coded. If the encoding parameter informationobtained by performing lossless decoding is intra predictioninformation, the lossless decoding unit 52 supplies the encodingparameter information to the intra prediction unit 62, and moves on tostep ST132. If the encoding parameter information is not intraprediction information, the lossless decoding unit 52 supplies theencoding parameter information to the motion compensation unit 63, andmoves on to step ST133.

In step ST132, the intra prediction unit 62 performs intra predictionoperations. Using decoded image data supplied from the addition unit 55and the encoding parameter information, the intra prediction unit 62performs intra predictions, to generate predicted image data.

In step ST133, the motion compensation unit 63 performs inter predictionoperations. Based on the encoding parameter information and motionvectors supplied from the lossless decoding unit 52, the motioncompensation unit 63 performs motion compensation on decoded image datasupplied from the frame memory 61. Further, the motion compensation unit63 outputs the predicted image data generated through the motioncompensation to the selector 64.

As described above, by the image decoding apparatus and method of thepresent invention, decoding is performed on an encoded bit streamgenerated by processing coefficient data obtained through orthogonaltransforms using bases that are set beforehand in accordance with blocklocations. In this decoding, inverse orthogonal transforms are performedby using bases that are set beforehand in accordance with the blocklocations in the macroblock indicated by the encoding parameterinformation contained in the encoded bit stream. Coefficient datasubjected to orthogonal transforms can be returned to the predictionerror data not yet subjected to the orthogonal transforms. Accordingly,even after orthogonal transforms using the bases compatible with theblock locations in the macroblock are performed, the resultant data canbe returned to the prediction error data not yet subjected to theorthogonal transforms. Also, even after encoding is performed by usingthe base in accordance with the prediction mode, the coefficient datasubjected to orthogonal transforms can be returned to the predictionerror data not yet subjected to the orthogonal transforms by using thebase that is set beforehand in accordance with the prediction modeindicated by the encoding parameter information.

<7. Base Learning Operation>

Next, a base generation unit that generates the bases to be used in theorthogonal transform unit 14 and the inverse orthogonal transform unit54 through a learning operation is described. FIG. 19 is a flowchartshowing the base learning operation, and the base generation unitgenerates the bases by performing the operation shown in FIG. 19 usingimages prepared for the learning. As the images for the learning, asmany different images as possible are used so that biases do not occurin the learning depending on the contents of the images.

In step ST141, the base generation unit determines whether there is animage not yet used for the learning. If there is an image not yet usedfor the learning, the base generation unit moves on to step ST142. Ifall the images have been used for the learning, the base generation unitmoves on to step ST152.

In step ST142, the base generation unit determines whether there is amacroblock not yet used for the learning. If there is a macroblock notyet used for the learning among the images prepared for the learning,the base generation unit moves on to step ST143. If all the macroblockshave been used for the learning, the base generation unit returns tostep ST141.

In step ST143, the base generation unit determines whether themacroblock size is 16×16 pixels. If the macroblock size is 16×16 pixels,the base generation unit moves on to step ST144, and if not 16×16pixels, moves on to step ST148.

In step ST144, the base generation unit generates 16×16 prediction errordata. The base generation unit performs intra predictions, to generatethe prediction error data of 16×16 pixels.

In step ST145, the base generation unit calculates a symmetric matrix ofa 4×4 orthogonal transform. The base generation unit divides the 16×16prediction error data into sixteen transform blocks of 4×4 pixels, andcalculates a symmetric matrix M for each prediction mode and each of theblock locations of the transform blocks in the macroblock. The basegeneration unit arranges the prediction error data of the transformblocks of 4×4 pixels into 16-order vectors, and calculates thedifference between the average of the 16-order vectors and each vector.With the difference being “q”, the base generation unit performs thecalculation expressed by the equation (3), to determine the symmetricmatrix M.

$\begin{matrix}\left\lbrack {{Mathematical}\mspace{14mu}{Formula}\mspace{14mu} 1} \right\rbrack & \; \\{M_{({{mdt},{mdi},{loc}})} = {\sum\limits_{\alpha = {num}}^{\;}\;{q_{\alpha}q_{\alpha}^{T}}}} & (3)\end{matrix}$

In the equation (3), “mdt” represents transform mode information fromwhich the macroblock size and the transform block size can bedetermined. In the above equation, “mid” represents the prediction modeof the intra predictions. In the above equation, “loc” represents theblock location of the transform block in the macroblock. In the aboveequation, “num” represents the number of learning times. Also, in theabove equation, “T” indicates that the matrix is a transposed matrix.

In step ST146, the base generation unit calculates a symmetric matrix ofan 8×8 orthogonal transform. The base generation unit divides the 16×16prediction error data into four transform blocks of 8×8 pixels, andcalculates a symmetric matrix M for each prediction mode and each of theblock locations of the transform blocks in the macroblock. The basegeneration unit arranges the prediction error data of the transformblocks of 8×8 pixels into 64-order vectors, and calculates thedifference between the average of the 64-order vectors and each vector.With the difference being “q”, the base generation unit performs thecalculation expressed by the equation (3), to determine the symmetricmatrix M.

In step ST147, the base generation unit calculates a symmetric matrix ofa 16×16 orthogonal transform. The base generation unit arranges theprediction error data of a transform block of 16×16 pixels into256-order vectors for each prediction mode, and calculates thedifference between the average of the 256-order vectors and each vector.With the difference being “q”, the base generation unit performs thecalculation expressed by the equation (3), to determine the symmetricmatrix M for each prediction mode.

Moving from step ST143 on to step ST148, the base generation unitdetermines whether the macroblock size is 8×8 pixels. If the macroblocksize is 8×8 pixels, the base generation unit moves on to step ST149, andif not 8×8 pixels, returns to step ST142.

In step ST149, the base generation unit generates 8×8 prediction errordata. The base generation unit performs intra predictions, to generatethe prediction error data of 8×8 pixels.

In step ST150, the base generation unit calculates a symmetric matrix ofa 4×4 orthogonal transform. The base generation unit divides the 8×8prediction error data into four transform blocks of 4×4 pixels, andcalculates a symmetric matrix M for each prediction mode and each of theblock locations of the transform blocks in the macroblock. The basegeneration unit arranges the prediction error data of the transformblocks of 4×4 pixels into 16-order vectors, and calculates thedifference between the average of the 16-order vectors and each vector.With the difference being “q”, the base generation unit performs thecalculation expressed by the equation (3), to determine the symmetricmatrix M.

In step ST151, the base generation unit calculates a symmetric matrix ofan 8×8 orthogonal transform. The base generation unit arranges theprediction error data of the transform blocks of 8×8 pixels into64-order vectors for each prediction mode, and calculates the differencebetween the average of the 64-order vectors and each vector. With thedifference being “q”, the base generation unit performs the calculationexpressed by the equation (3), to determine the symmetric matrix M foreach prediction mode.

In step ST152, the base generation unit calculates bases of KLtransforms. The base generation unit determines the eigenvectorscorresponding to the eigenvalues of the respective symmetric matrixes M,and arranges the eigenvectors in order of eigenvalue size, to generatethe bases of KL transforms.

By performing the above operation, the bases to be used in the KLtransforms by the 16×16 KL transform unit 141, the 8×8 KL transform unit142, the 2×2 KL transform units 143 and 146, and the 4×4 KL transformunits 144 and 145 can be generated. Also, by calculating the inversematrixes of the respective bases, the bases to be used in the KL inversetransforms by the 16×16 KL inverse transform unit 541, the 2×2 KLinverse transform units 542 and 545, the 8×8 KL inverse transform unit543, and the 4×4 KL inverse transform units 544 and 546 can begenerated.

Further, if the bases to be used for performing KL transforms and KLinverse transforms on the respective blocks for the respective blocklocations in each of the macroblock sizes and for each of the predictionmodes are stored in the image encoding apparatus and the image decodingapparatus, the number of bases to be stored becomes large. That is, amemory with a large capacity is required. In view of this, the bases aredivided into groups, to reduce the bases to be stored.

Next, two example methods of grouping are described. By a first method,Euclidean distances between bases learned through learning arecalculated, and the bases between which the distances are short aregrouped together. The bases in the group are replaced with one baserepresenting the group. The number of bases can be reduced by performinggrouping in this manner.

By a second method, grouping is performed in accordance with distancesfrom a reference pixel. As shown in FIG. 20, in the prediction mode 0(vertical), the blocks of Group 1={P4, P5, P6, P7} are at the samedistance from the reference pixel. In this case, the prediction errorsof the pixels P4, P5, P6, and P7 often have similar characteristics.Therefore, one base is used for Group 1. Likewise, one base is used forGroup 0, one base is used for Group 2, and one base is used for Group 3.Accordingly, the number of bases can be reduced from 16 to 4.

Likewise, in the prediction mode 1 (horizontal), the blocks of Group1={P1, P5, P9, P13} are in the same positional relationship with (or atthe same distance from) the reference pixel. In this case, theprediction errors of the pixels P1, P5, P9, and P13 often have similarcharacteristics. Therefore, one base is used for Group 1. Likewise, onebase is used for Group 0, one base is used for Group 2, and one base isused for Group 3. Accordingly, the number of bases can be reduced from16 to 4.

In the prediction mode 4 (diagonal down-right), the relationshipsbetween the reference pixel and the respective blocks are not the same.When rotated 90 degrees, however, P12 has the same positionalrelationship with the reference pixel as P3. In view of this, the pixelsare grouped into {P1, P4}, {P2, P8}, {P6, P9}, {P7, P13}, and {P11, P14}in each of which the blocks have the same positional relationship withthe reference pixel when rotated 90 degrees, and one base is used forone group.

Further, when rotated 90 degrees in the prediction mode 0 (vertical),the respective blocks have the same positional relationships with thereference pixel as those in the prediction mode 1 (horizontal).Accordingly, the prediction mode 0 (vertical) and the prediction mode 1(horizontal) are grouped together, so that the number of bases can befurther reduced.

<8. Case of Software Processing>

The series of operations described in this specification can beperformed by hardware, software, or a combination of hardware andsoftware. In a case where operations are performed by software, aprogram in which the operation sequences are recorded is installed in amemory incorporated into specialized hardware in a computer.Alternatively, the operations can be performed by installing the programinto a general-purpose computer that can perform various kinds ofoperations.

For example, the program can be recorded beforehand on a hard disk or aROM (Read Only Memory) as a recording medium. Alternatively, the programcan be temporarily or permanently stored (recorded) on a removablerecording medium such as a flexible disk, a CD-ROM (Compact Disc ReadOnly Memory), a MO (MagnetoOptical) disk, a DVD (Digital VersatileDisc), a magnetic disk, or a semiconductor memory. Such a removablerecording medium can be provided as so-called package software.

The program can be not only installed into the computer from the abovedescribed removable recording medium, but also wirelessly transferredfrom a download site to the computer or transferred to the computer bywire via a LAN (Local Area Network) or the Internet so that the programtransferred in this manner can be received in the computer and beinstalled into a recording medium such as an internal hard disk.

The step of writing the program includes operations to be performed inparallel or independently of one another if not necessarily inchronological order, as well as operations to be performed inchronological order in accordance with specified order.

<9. Case of Application to an Electronic Device>

In the above described examples, H.264/AVC is used as theencoding/decoding technique. However, the present invention can beapplied to image encoding apparatuses and image decoding apparatusesthat use other encoding/decoding techniques.

Further, the present invention can be applied to image encodingapparatuses and image decoding apparatuses that are used when imageinformation (encoded bit streams) compressed through orthogonaltransforms such as discrete cosine transforms and motion compensation asin MPEG or H.26× is received via a network medium such as satellitebroadcasting, cable TV (television), the Internet, or a portabletelephone device, or is processed in a storage medium such as an opticalor magnetic disk or a flash memory, for example.

The above described image encoding apparatus 10 and the image decodingapparatus 50 can be applied to any electronic apparatuses. The followingis a description of such examples.

FIG. 21 schematically shows an example structure of a televisionapparatus to which the present invention is applied. The televisionapparatus 90 includes an antenna 901, a tuner 902, a demultiplexer 903,a decoder 904, a video signal processing unit 905, a display unit 906,an audio signal processing unit 907, a speaker 908, and an externalinterface unit 909. The television apparatus 90 further includes acontrol unit 910, a user interface unit 911, and the like.

The tuner 902 selects a desired channel from broadcast wave signalsreceived at the antenna 901, and performs demodulation. The resultantencoded bit stream is output to the demultiplexer 903.

The demultiplexer 903 extracts the video and audio packets of the showto be viewed from the encoded bit stream, and outputs the data of theextracted packet to the decoder 904. The demultiplexer 903 also suppliesa packet of data such as EPG (Electronic Program Guide) to the controlunit 910. Where scrambling is performed, the demultiplexer or the likecancels the scrambling.

The decoder 904 performs a packet decoding operation, and outputs thevideo data generated through the decoding operation to the video signalprocessing unit 905, and the audio data to the audio signal processingunit 907.

The video signal processing unit 905 subjects the video data to a noiseremoval and video processing or the like in accordance with usersettings. The video signal processing unit 905 generates video data ofthe show to be displayed on the display unit 906, or generates imagedata or the like through an operation based on an application suppliedvia a network. The video signal processing unit 905 also generates videodata for displaying a menu screen or the like for item selection, andsuperimposes the video data on the video data of the show. Based on thevideo data generated in this manner, the video signal processing unit905 generates a drive signal to drive the display unit 906.

Based on the drive signal from the video signal processing unit 905, thedisplay unit 906 drives a display device (a liquid crystal displayelement, for example) to display the video of the show.

The audio signal processing unit 907 subjects the audio data topredetermined processing such as a noise removal, and performs a D/Aconversion operation and an amplification operation on the processedaudio data. The resultant audio data is supplied as an audio output tothe speaker 908.

The external interface unit 909 is an interface for a connection with anexternal device or a network, and transmits and receives data such asvideo data and audio data.

The user interface unit 911 is connected to the control unit 910. Theuser interface unit 911 is formed with operation switches, a remotecontrol signal reception unit, and the like, and supplies an operatingsignal according to a user operation to the control unit 910.

The control unit 910 is formed with a CPU (Central Processing Unit), amemory, and the like. The memory stores the program to be executed bythe CPU, various kinds of data necessary for the CPU to performoperations, the EPG data, data obtained via a network, and the like. Theprogram stored in the memory is read and executed by the CPU at apredetermined time such as the time of activation of the televisionapparatus 90. The CPU executes the program to control the respectivecomponents so that the television apparatus 90 operates in accordancewith a user operation.

In the television apparatus 90, a bus 912 is provided for connecting thetuner 902, the demultiplexer 903, the video signal processing unit 905,the audio signal processing unit 907, the external interface unit 909,and the like to the control unit 910.

In the television apparatus having such a structure, the decoder 904 hasthe functions of the image decoding apparatus (the image decodingmethod) of the present invention. As the functions of the image encodingapparatus of the present invention are used on the broadcasting stationside, encoded bit streams can be correctly decoded in the televisionapparatus, even if the encoded bit streams have been generated at ahigher encoding efficiency and with higher image quality.

FIG. 22 schematically shows an example structure of a portable telephonedevice to which the present invention is applied. The portable telephonedevice 92 includes a communication unit 922, an audio codec 923, acamera unit 926, an image processing unit 927, a demultiplexing unit928, a recording/reproducing unit 929, a display unit 930, and a controlunit 931. Those components are connected to one another via a bus 933.

Also, an antenna 921 is connected to the communication unit 922, and aspeaker 924 and a microphone 925 are connected to the audio codec 923.Further, an operation unit 932 is connected to the control unit 931.

The portable telephone device 92 performs various kinds of operationssuch as transmission and reception of audio signals, transmission andreception of electronic mail and image data, image capturing, and datarecording, in various kinds of modes such as an audio communication modeand a data communication mode.

In the audio communication mode, an audio signal generated at themicrophone 925 is converted into audio data, and the data is compressedat the audio codec 923. The compressed data is supplied to thecommunication unit 922. The communication unit 922 performs a modulationoperation, a frequency conversion operation, and the like on the audiodata, to generate a transmission signal. The communication unit 922 alsosupplies the transmission signal to the antenna 921, and thetransmission signal is transmitted to a base station (not shown). Thecommunication unit 922 also amplifies a signal received at the antenna921, and performs a frequency conversion operation, a demodulationoperation, and the like. The resultant audio data is supplied to theaudio codec 923. The audio codec 923 decompresses audio data, andconverts the audio data into an analog audio signal that is output tothe speaker 924.

In a case where mail transmission is performed in the data communicationmode, the control unit 931 receives text data that is input through anoperation by the operation unit 932, and the input text is displayed onthe display unit 930. In accordance with a user instruction or the likethrough the operation unit 932, the control unit 931 generates andsupplies mail data to the communication unit 922. The communication unit922 performs a modulation operation, a frequency conversion operation,and the like on the mail data, and transmits the resultant transmissionsignal from the antenna 921. The communication unit 922 also amplifies asignal received at the antenna 921, and performs a frequency conversionoperation, a demodulation operation, and the like, to restore the maildata. This mail data is supplied to the display unit 930, and the mailcontent is displayed.

The portable telephone device 92 can cause the recording/reproducingunit 929 to store received mail data into a storage medium. The storagemedium is a rewritable storage medium. For example, the storage mediummay be a semiconductor memory such as a RAM or an internal flash memory,a hard disk, or a removable medium such as a magnetic disk, amagnetooptical disk, an optical disk, a USB memory, or a memory card.

In a case where image data is transmitted in the data communicationmode, image data generated at the camera unit 926 is supplied to theimage processing unit 927. The image processing unit 927 performs anencoding operation on the image data, to generate encoded data.

The demultiplexing unit 928 multiplexes the encoded data generated atthe image processing unit 927 and the audio data supplied from the audiocodec 923 by a predetermined technique, and supplies the multiplexeddata to the communication unit 922. The communication unit 922 performsa modulation operation, a frequency conversion operation, and the likeon the multiplexed data, and transmits the resultant transmission signalfrom the antenna 921. The communication unit 922 also amplifies a signalreceived at the antenna 921, and performs a frequency conversionoperation, a demodulation operation, and the like, to restore themultiplexed data. This multiplexed data is supplied to thedemultiplexing unit 928. The demultiplexing unit 928 divides themultiplexed data, and supplies the encoded data to the image processingunit 927, and the audio data to the audio codec 923. The imageprocessing unit 927 performs a decoding operation on the encoded data,to generate image data. This image data is supplied to the display unit930, to display the received images. The audio codec 923 converts theaudio data into an analog audio signal that is output to the speaker924, and the received sound is output.

In the portable telephone device having the above structure, the imageprocessing unit 927 has the functions of the image encoding apparatus(the image encoding method) and the image decoding apparatus (the imagedecoding method) of the present invention. Accordingly, when image datais communicated, encoding efficiency and image quality can be improved.

FIG. 23 schematically shows an example structure of arecording/reproducing apparatus to which the present invention isapplied. The recording/reproducing apparatus 94 records the audio dataand video data of a received broadcast show on a recording medium, andprovides the recorded data to a user at a time according to aninstruction from the user. The recording/reproducing apparatus 94 canalso obtain audio data and video data from another apparatus, forexample, and record the data on a recording medium. Further, therecording/reproducing apparatus 94 decodes and outputs audio data andvideo data recorded on a recording medium, so that a monitor device orthe like can display images and outputs sound.

The recording/reproducing apparatus 94 includes a tuner 941, an externalinterface unit 942, an encoder 943, a HDD (Hard Disk Drive) unit 944, adisk drive 945, a selector 946, a decoder 947, an OSD (On-ScreenDisplay) unit 948, a control unit 949, and a user interface unit 950.

The tuner 941 selects a desired channel from broadcast signals receivedat an antenna (not shown). The tuner 941 demodulates the received signalof the desired channel, and outputs the resultant encoded bit stream tothe selector 946.

The external interface unit 942 is formed with at least one of anIEEE1394 interface, a network interface unit, a USB interface, a flashmemory interface, and the like. The external interface unit 942 is aninterface for a connection with an external device, a network, a memorycard, or the like, and receives data such as video data and audio datato be recorded and the like.

The encoder 943 performs predetermined encoding on video data and audiodata that are supplied from the external interface unit 942 and are notencoded, and outputs the encoded bit stream to the selector 946.

The HDD unit 944 records content data such as videos and sound, variouskinds of programs, other data, and the like on an internal hard disk,and reads the data from the hard disk at the time of reproduction or thelike.

The disk drive 945 performs signal recording and reproduction on amounted optical disk. The optical disk may be a DVD disk (such as aDVD-Video, a DVD-RAM, a DVD-R, a DVD-RW, a DVD+R, or a DVD+RW) or aBlu-ray disk, for example.

The selector 946 selects an encoded bit stream from the tuner 941 or theencoder 943 at the time of video and audio recording, and supplies theencoded bit stream to either the HDD unit 944 or the disk drive 945. Theselector 946 also supplies an encoded bit stream output from the HDDunit 944 or the disk drive 945 to the decoder 947 at the time of videoand audio reproduction.

The decoder 947 performs a decoding operation on an encoded bit stream.The decoder 947 supplies the video data generated by performing thedecoding to the OSD unit 948. The decoder 947 also outputs the audiodata generated by performing the decoding.

The OSD unit 948 also generates video data for displaying a menu screenor the like for item selection, and superimposes the video data on videodata output from the decoder 947.

The user interface unit 950 is connected to the control unit 949. Theuser interface unit 950 is formed with operation switches, a remotecontrol signal reception unit, and the like, and supplies an operatingsignal according to a user operation to the control unit 949.

The control unit 949 is formed with a CPU, a memory, and the like. Thememory stores the program to be executed by the CPU and various kinds ofdata necessary for the CPU to perform operations. The program stored inthe memory is read and executed by the CPU at a predetermined time suchas the time of activation of the recording/reproducing apparatus 94. TheCPU executes the program to control the respective components so thatthe recording/reproducing apparatus 94 operates in accordance with auser operation.

In the recording/reproducing apparatus having the above structure, theencoder 943 has the functions of the image encoding apparatus (the imageencoding method) of the present invention, and the decoder 947 has thefunctions of the image decoding apparatus (the image decoding method),so that encoding efficiency and image quality can be improved, and videorecording and reproduction can be efficiently performed.

FIG. 24 schematically shows an example structure of an imaging apparatusto which the present invention is applied. An imaging apparatus 96captures an image of an object, and causes a display unit to display theimage of the object or records the image as image data on a recordingmedium.

The imaging apparatus 96 includes an optical block 961, an imaging unit962, a camera signal processing unit 963, an image data processing unit964, a display unit 965, an external interface unit 966, a memory unit967, a media drive 968, an OSD unit 969, and a control unit 970. A userinterface unit 971 is connected to the control unit 970. Further, theimage data processing unit 964, the external interface unit 966, thememory unit 967, the media drive 968, the OSD unit 969, the control unit970, and the like are connected via a bus 972.

The optical block 961 is formed with a focus lens, a diaphragm, and thelike. The optical block 961 forms an optical image of an object on theimaging surface of the imaging unit 962. Formed with a CCD or a CMOSimage sensor, the imaging unit 962 generates an electrical signal inaccordance with the optical image through a photoelectric conversion,and supplies the electrical signal to the camera signal processing unit963.

The camera signal processing unit 963 performs various kinds of camerasignal processing such as a knee correction, a gamma correction, and acolor correction on the electrical signal supplied from the imaging unit962. The camera signal processing unit 963 supplies the image datasubjected to the camera signal processing to the image data processingunit 964.

The image data processing unit 964 performs an encoding operation on theimage data supplied from the camera signal processing unit 963. Theimage data processing unit 964 supplies the encoded data generated byperforming the encoding operation to the external interface unit 966 andthe media drive 968. The image data processing unit 964 also performs adecoding operation on the encoded data supplied from the externalinterface unit 966 and the media drive 968. The image data processingunit 964 supplies the image data generated by performing the decodingoperation to the display unit 965. The image data processing unit 964also performs an operation to supply the image data supplied from thecamera signal processing unit 963 to the display unit 965, orsuperimposes display data obtained from the OSD unit 969 on the imagedata and supplies the image data to the display unit 965.

The OSD unit 969 generates a menu screen formed with symbols,characters, or figures, or display data such as icons, and outputs thedata to the image data processing unit 964.

The external interface unit 966 is formed with a USB input/outputterminal and the like, for example, and is connected to a printer whenimage printing is performed. A drive is also connected to the externalinterface unit 966 where necessary, and a removable medium such as amagnetic disk or an optical disk is mounted on the drive as appropriate.A computer program read from such a removable disk is installed wherenecessary. Further, the external interface unit 966 includes a networkinterface connected to a predetermined network such as a LAN or theinternet. The control unit 970 reads encoded data from the memory unit967 in accordance with an instruction from the user interface unit 971,for example, and can supply the encoded data from the external interfaceunit 966 to another apparatus connected via a network. The control unit970 can also obtain, via the external interface unit 966, encoded dataor image data supplied from another apparatus via a network, and supplythe encoded data or image data to the image data processing unit 964.

A recording medium to be driven by the media drive 968 may be areadable/rewritable removable disk such as a magnetic disk, amagnetooptical disk, an optical disk, or a semiconductor memory. Therecording medium may be any type of removable medium, and may be a tapedevice, a disk, or a memory card. The recording medium may of course bea non-contact IC card or the like.

Alternatively, the media drive 968 and a recording medium may beintegrated, and may be formed with an immobile storage medium such as aninternal hard disk drive or a SSD (Solid State Drive).

The control unit 970 is formed with a CPU, a memory, and the like. Thememory stores the program to be executed by the CPU, various kinds ofdata necessary for the CPU to perform operations, and the like. Theprogram stored in the memory is read and executed by the CPU at apredetermined time such as the time of activation of the imagingapparatus 96. The CPU executes the program to control the respectivecomponents so that the imaging apparatus 96 operates in accordance witha user operation.

In the imaging apparatus having the above structure, the image dataprocessing unit 964 has the functions of the image encoding apparatus(the image encoding method) and the image decoding apparatus (the imagedecoding method) of the present invention. Accordingly, when capturedimages are recorded into the memory unit 967 or a recording medium orthe like, encoding efficiency and image quality can be improved, and thecaptured images can be efficiently recorded and reproduced.

Further, the present invention should not be interpreted to be limitedto the above described embodiments of the invention. For example, thepresent invention should not be limited to the above describedmacroblock sizes, transform block sizes, and prediction modes. Theembodiments of the invention disclose the present invention throughexamples, and it should be obvious that those skilled in the art canmodify or replace those embodiments with other embodiments withoutdeparting from the scope of the invention. That is, the claims should betaken into account in understanding the subject matter of the invention.

Industrial Applicability

With the image decoding apparatus, the image encoding apparatus, and themethods and programs of the present invention, the bases that are setbeforehand in accordance with the block locations of the transformblocks in a macroblock are used to perform orthogonal transforms at thetime of image data encoding. Also, in decoding an encoded bit streamthat is generated by processing coefficient data obtained through anorthogonal transform using the bases set beforehand in accordance withthe block locations, the bases that are set beforehand in accordancewith the block locations in the macroblock indicated by the encodingparameter information contained in the encoded bit stream are used toperform an inverse orthogonal transform, and the coefficient datasubjected to the orthogonal transform is returned to the predictionerror data not yet subjected to the orthogonal transform. As anorthogonal transform and an inverse orthogonal transform are performedby using the bases compatible with the block locations in a macroblockas described above, transforms that are optimized in accordance with theblock locations can be performed, and encoding efficiency can beincreased. Accordingly, the present invention is suitably applied to animage decoding apparatus, an image encoding apparatus, and the like thatare used when image information (encoded bit streams) obtained throughblock-based encoding as in MPEG, H.26×, or the like is transmitted orreceived via a network medium such as satellite broadcasting, cable TV,the Internet, or a portable telephone device.

REFERENCE SIGNS LIST

-   10 . . . image encoding apparatus 11 . . . A/D conversion unit 12,    57 . . . picture rearrangement buffer 13 . . . subtraction unit 14 .    . . orthogonal transform unit 15 . . . quantization unit 16 . . .    lossless coding unit 17, 51 . . . accumulation buffer 18 . . . rate    control unit 21, 53 . . . inverse quantization unit 22, 54 . . .    inverse orthogonal transform unit 23, 55 . . . addition unit 24, 56    . . . deblocking filter 27, 61 . . . frame memory 31, 62 . . . intra    prediction unit 32, 63 . . . motion prediction/compensation unit 33    . . . predicted image/optimum mode select unit 50 . . . image    decoding apparatus 52 . . . lossless decoding unit 58 . . . D/A    conversion unit 64, 946 . . . selector 90 . . . television apparatus    92 . . . portable telephone device 94 . . . recording/reproducing    apparatus 96 . . . imaging apparatus 141 . . . 16×16 KL transform    unit 142 . . . 8×8 KL transform unit 143, 146 . . . 2×2 KL transform    unit 144, 145 . . . 4×4 KL transform unit 147 . . . DCT unit 148 . .    . coefficient select unit 541 . . . 16×16 KL inverse transform unit    542, 545 . . . 2×2 KL inverse transform unit 543 . . . 8×8 KL    inverse transform unit 544, 546 . . . KL inverse transform unit 547    . . . IDCT unit 548 . . . data select unit 901, 921 . . . antenna    902, 941 . . . tuner 903 . . . demultiplexer 904, 947 . . . decoder    905 . . . video signal processing unit 906 . . . display unit 907 .    . . audio signal processing unit 908 . . . speaker 909, 942, 966 . .    . external interface unit 910, 931, 949, 970 . . . control unit 911,    932, 971 . . . user interface unit 912, 933, 972 . . . bus 922 . . .    communication unit 923 . . . audio codec 924 . . . speaker 925 . . .    microphone 926 . . . camera unit 927 . . . image processing unit 928    . . . demultiplexing unit 929 . . . recording/reproducing unit 930 .    . . display unit 943 . . . encoder 944 . . . HDD unit 945 . . . disk    drive 948, 969 . . . OSD unit 961 . . . optical block 962 . . .    imaging unit 963 . . . camera signal processing unit 964 . . . image    data processing unit 965 . . . display unit 967 . . . memory unit    968 . . . media drive

The invention claimed is:
 1. An image decoding apparatus that performsan orthogonal transform on prediction error data of each transformblock, and decodes image data from an encoded bit stream generated byprocessing coefficient data subjected to the orthogonal transform, theprediction error data being a difference between the image data andpredicted image data, the image decoding apparatus comprising: a dataprocessing unit configured to process the encoded bit stream to obtainthe coefficient data subjected to the orthogonal transform and encodingparameter information; a selector to determine an inverse orthogonaltransform procedure based on a size of a transform block and aprediction mode; an inverse orthogonal transform unit configured toperform the determined inverse orthogonal transform procedure on thecoefficient data by using a base to obtain the prediction error data,the base being set beforehand in accordance with a location of thetransform block in a macroblock indicated by the encoding parameterinformation; a predicted image data generation unit configured togenerate the predicted image data; and an addition unit configured toadd the predicted image data generated by the predicted image datageneration unit to the prediction error data obtained by the inverseorthogonal transform unit, to decode the image data.
 2. The imagedecoding apparatus according to claim 1, wherein the inverse orthogonaltransform unit performs the inverse orthogonal transform by using a baseset beforehand in accordance with the location of the transform blockand a prediction mode indicated by the encoding parameter information.3. The image decoding apparatus according to claim 2, wherein, when theencoding parameter information indicates that the macroblock contains aplurality of transform blocks, the inverse orthogonal transform unitperforms the inverse orthogonal transform by using a base onorthogonally transformed coefficient data of the lowest-frequencycomponent coefficient data after an orthogonal transform of therespective transform blocks contained in the macroblock, the base beingset beforehand in accordance with the prediction mode.
 4. The imagedecoding apparatus according to claim 2, wherein the base used by theinverse orthogonal transform unit is an inverse matrix of a base used inthe orthogonal transform of the prediction error data of each transformblock.
 5. The image decoding apparatus according to claim 1, wherein theinverse orthogonal transform unit performs a Karhunen-Loeve inversetransform using the base.
 6. An image decoding method for performing anorthogonal transform on prediction error data of each transform block,and decoding image data from an encoded bit stream generated byprocessing coefficient data subjected to the orthogonal transform, theprediction error data being a difference between the image data andpredicted image data, the image decoding method comprising the steps of:processing the encoded bit stream to obtain the coefficient datasubjected to the orthogonal transform and encoding parameterinformation; determine an inverse orthogonal transform procedure basedon a size of a transform block and a prediction mode; performing thedetermined inverse orthogonal transform procedure on the coefficientdata by using a base to obtain the prediction error, the base being setbeforehand in accordance with a location of the transform block in amacroblock indicated by the encoding parameter information; generatingthe predicted image data; and adding the generated predicted image datato the prediction error obtained by the inverse orthogonal transformunit, to decode the image data.
 7. A non-transitory computer-mediumstoring a program for causing a computer to perform an orthogonaltransform on prediction error data of each transform block, and decodeimage data from an encoded bit stream generated by processingcoefficient data subjected to the orthogonal transform, the predictionerror data being a difference between the image data and predicted imagedata, the program causing the computer to: process the encoded bitstream to obtain the coefficient data subjected to the orthogonaltransform and encoding parameter information; determine an inverseorthogonal transform procedure based on a size of a transform block anda prediction mode; perform the determined inverse orthogonal transformprocedure on the coefficient data by using a base to obtain theprediction error, the base being set beforehand in accordance with alocation of the transform block in a macroblock indicated by theencoding parameter information; generate the predicted image data; andadd the generated predicted image data to the prediction error obtainedby the inverse orthogonal transform unit, to decode the image data. 8.An image encoding apparatus that encodes image data, the image encodingapparatus comprising: a prediction unit configured to generate predictedimage data of the image data; a subtraction unit configured to generateprediction error data, the prediction error data being a differencebetween the image data and the predicted image data; a selector todetermine an orthogonal transform procedure based on a size of atransform block and a prediction mode; an orthogonal transform unitconfigured to perform the determined orthogonal transform procedure onthe prediction error of each transform block by using a base setbeforehand in accordance with a location of the transform block in amacroblock; and a data processing unit configured to process data outputfrom the orthogonal transform unit, to generate an encoded bit stream.9. The image encoding apparatus according to claim 8, wherein theorthogonal transform unit performs the orthogonal transform by using abase set beforehand in accordance with the location of the transformblock and a prediction mode used when the prediction unit generates thepredicted image data.
 10. The image encoding apparatus according toclaim 9, wherein, when the macroblock contains a plurality of transformblocks, the orthogonal transform unit performs an orthogonal transformby using a base, the orthogonal transform being performed on a blockusing the lowest-frequency component coefficient among the respectivetransform blocks subjected to the orthogonal transform in themacroblock, the base being set beforehand in accordance with theprediction mode.
 11. The image encoding apparatus according to claim 9,wherein the base used by the orthogonal transform unit is eigenvectorscorresponding to eigenvalues of a matrix calculated from a size of themacroblock, a size of the transform block, locations of the transformblocks in the macroblock, and predicted error data in each transformblock in each prediction mode.
 12. The image encoding apparatusaccording to claim 11, wherein the base used by the orthogonal transformunit is grouped in accordance with a distance between bases.
 13. Theimage encoding apparatus according to claim 11, wherein the base used bythe orthogonal transform unit is grouped in accordance with a distancefrom a reference pixel.
 14. The image encoding apparatus according toclaim 8, wherein the orthogonal transform unit performs a Karhunen-Loevetransform by using the base.
 15. An image encoding method for encodingimage data, the image encoding method comprising the steps of:generating predicted image data of the image data; generating predictionerror data, the prediction error data being a difference between theimage data and the predicted image data; determining an orthogonaltransform procedure based on a size of a transform block and aprediction mode; and performing the determined orthogonal transformprocedure on the prediction error of each transform block by using abase set beforehand in accordance with a location of the transform blockin a macroblock.
 16. A non-transitory computer-medium storing a programfor causing a computer to encode image data, the program causing thecomputer to: generate predicted image data of the image data; generateprediction error data, the prediction error data being a differencebetween the image data and the predicted image data; determine anorthogonal transform procedure based on a size of a transform block anda prediction mode; and perform the determined orthogonal transformprocedure on the prediction error of each transform block by using abase set beforehand in accordance with a location of the transform blockin a macroblock.